Kafka Streams Testing with Scala Part 1
After experimenting with Kafka Streams with Scala in the first post [1], I started to wonder how one goes about Kafka Streams testing in Java or Scala. How does one create and run automated tests for Kafka Streams applications? In this post, I’ll describe what I’ve learned so far. Also, if you have any questions or ideas […]
Kafka Streams with Scala for Beginners Example
If you’re new to Kafka Streams with Scala, here’s a quick start example which may help jumpstart your efforts. My plan is to keep updating the sample project, so let me know if you would like to see anything in particular with Kafka Streams with Scaa. In this example, the intention is to 1) provide a SBT project you can pull, […]
Apache Kafka Architecture – Delivery Semantics
Apache Kafka offers message delivery guarantees between producers and consumers. These delivery guarantees can be divided into three groups which include “at most once”, “at least once” and “exactly once”. at most once which can lead to messages being lost, but they cannot be redelivered or duplicated at least once ensures messages are never lost but may be duplicated […]
How to Debug Scala Spark in IntelliJ
Have you struggled to configure debugging in IntelliJ for your Spark programs? Yeah, me too. Debugging with Scala code was easy, but when I moved to Spark things didn’t work as expected. So, in this tutorial, let’s cover debugging Scala based Spark programs in IntelliJ tutorial. We’ll go through a few examples and utilize the occasional help […]
Change Data Capture – What, Why, How
Change Data Capture Overview Change Data Capture (CDC) refers to the determination and tracking of changes made to the data in databases. CDC is utilized to extract and transport changes in online transactional processing (OLTP) data systems to downstream systems such as data warehouses, data lakes and/or stream processors. In essence, CDC is implemented by writing to […]
Kafka Connect mySQL Examples
In this Kafka Connect mysql tutorial, we’ll cover reading from mySQL to Kafka and reading from Kafka and writing to mySQL. Let’s run this on your environment. Now, it’s just an example and we’re not going to debate operations concerns such as running in standalone or distributed mode. The focus will be keeping it simple and get it working. We can […]
Spark Kinesis Example – Moving Beyond Word Count
If you are looking for Spark with Kinesis example, you are in the right place. This Spark Kinesis tutorial intends to help you become better at integrating the two. In this post, I’m going to provide a custom Spark Kinesis code example and a screencast of running it. We’re going to cover running, configuring, sending […]
Recent Posts
- Kafka Streams Testing with Scala Part 1 January 7, 2019
- Kafka Streams with Scala for Beginners Example January 2, 2019
- Apache Kafka Architecture – Delivery Semantics December 11, 2018
Most Commented