Data Streaming 101 (and Real-Time Data Processing)

Data Streaming 101

Data streaming is the process of continuously transmitting data from a source to a destination in real-time. It can be a method for transmitting large amounts of data quickly and efficiently vs a more traditional method of accumulating data over time, and then transmitting in scheduled batches. As with most options in software architecture, there … Read more

Streaming Analytics in 2023 – What, Why, and How

streaming analytics - what, why and how

Streaming analytics continues to become more important because it lets businesses learn new things and make decisions in almost real time. This is especially relevant in fields like finance, health care, and manufacturing where the amount of time needed to make decisions is very critical. By the way, when measuring time in streaming analytics, you’ll … Read more

Open Source Change Data Capture in 2023

2023 Open Source Change Data Capture

Let’s consider three open source change data capture (CDC) options ready for production in the year 2023. Before we begin, let’s confirm we all see the CDC trend.  To me, it seems everywhere you look these days is all about change data capture. From my perspective that wasn’t the case for many years. Do you … Read more

Streaming Data Engineer Use Cases

Streaming Data Engineer Use Cases

As a streaming data engineer, we face many data integration challenges such as “How do we integrate this SaaS with this internal database?”, “Will a particular integration be real-time or batch?”, “How does the system we design recovery from possible failures?” and “If anyone has ever addressed a situation similar to mine before, how did … Read more

Schema Registry in Data Streaming [Options, Choices, Comparisons]

Schema Registry Options

A schema registry in data streaming use cases such as micro-service integration, streaming ETL, event driven architectures, log ingest stream processing, etc., is not a requirement, but there are numerous reasons for implementing one.  The reasoning for schema registries in data streaming architectures are plentiful and have been covered extensively already.  I’ve included some of … Read more

What and Why Event Logs?

Why Event Logs diagram

Before we begin diving into event logs, let’s start with a quote from one of my software heroes. “The idea of structuring data as a stream of events is nothing new, and it is used in many differentfields. Even though the underlying principles are often similar, the terminology is frequentlyinconsistent across different fields, which can … Read more

Stream Processing

Event Stream Processing

We choose Stream Processing as a way to process data more quickly than traditional approaches.  But, how do we do Stream Processing? Is Stream Processing different than Event Stream Processing?  Why do we need it?  What are a few examples of event streaming patterns?  How do we implement it? Let’s get into these questions. As … Read more

Stream Processor Windows

Stream Processor Windows

When moving to stream processing architecture or building stream processors, you will soon face two choices.  Will you process streams on an individual, per event basis?  Or, will you collect and buffer multiple events/messages first, and then apply a function or join results to this collection of events? Examples of single event processing might be … Read more

Change Data Capture – What Is It? How Does it Work?

Change Data Capture

Change Data Capture is a mechanism to capture the changes in databases so they may be processed someplace other than the database or application(s) which made the change.  This article will explain what change data capture (CDC) is, how it works, and why it’s important for businesses. Why?  Why would we want to capture changes … Read more