Apache Kafka Architecture – Delivery Semantics
Apache Kafka offers message delivery guarantees between producers and consumers. These delivery guarantees can be divided into three groups which include “at most once”, “at least once” and “exactly once”. at most once which can lead to messages being lost, but they cannot be redelivered or duplicated at least once ensures messages are never lost but may be duplicated […]
How to Debug Scala Spark in IntelliJ
Have you struggled to configure debugging in IntelliJ for your Spark programs? Yeah, me too. Debugging with Scala code was easy, but when I moved to Spark things didn’t work as expected. So, in this tutorial, let’s cover debugging Scala based Spark programs in IntelliJ tutorial. We’ll go through a few examples and utilize the occasional help […]
Change Data Capture – What, Why, How
Change Data Capture Overview Change Data Capture (CDC) refers to the determination and tracking of changes made to the data in databases. CDC is utilized to extract and transport changes in online transactional processing (OLTP) data systems to downstream systems such as data warehouses, data lakes and/or stream processors. In essence, CDC is implemented by writing to […]
Kafka Connect mySQL Examples
In this Kafka Connect mysql tutorial, we’ll cover reading from mySQL to Kafka and reading from Kafka and writing to mySQL. Let’s run this on your environment. Now, it’s just an example and we’re not going to debate operations concerns such as running in standalone or distributed mode. The focus will be keeping it simple and get it working. We can […]
Spark Kinesis Example – Moving Beyond Word Count
If you are looking for Spark with Kinesis example, you are in the right place. This Spark Kinesis tutorial intends to help you become better at integrating the two. In this post, I’m going to provide a custom Spark Kinesis code example and a screencast of running it. We’re going to cover running, configuring, sending […]
Spark Performance Monitoring Tools – A List of Options
Which spark performance monitoring tools are available to monitor the performance of your Spark cluster? Let’s find out. Before we address this question, I assume we already know Spark includes monitoring through the Spark UI. In addition, Spark includes support for monitoring and performance debugging through the History Server as well as support for the […]
Spark FAIR Scheduler Example
Scheduling in Spark can be a confusing topic. When someone says “scheduling” in spark, do they mean scheduling applications running on the same cluster? Or, do they mean the internal scheduling of Spark tasks within the Spark application? So, before we cover an example of utilizing the Spark FAIR Scheduler, let’s make sure we’re on the […]
Recent Posts
- Apache Kafka Architecture – Delivery Semantics December 11, 2018
- How to Debug Scala Spark in IntelliJ December 7, 2018
- Change Data Capture – What, Why, How December 4, 2018
Most Commented