How to Debug Scala Spark in IntelliJ

Have you struggled to configure debugging in IntelliJ for your Spark programs?  Yeah, me too.  Debugging with Scala code was easy, but when I moved to Spark things didn’t work as expected.  So, in this tutorial, let’s cover debugging Scala based Spark programs in IntelliJ tutorial.  We’ll go through a few examples and utilize the occasional help […]

3 days ago More
 

Change Data Capture – What, Why, How

Change Data Capture Overview Change Data Capture (CDC) refers to the determination and tracking of changes made to the data in databases. CDC is utilized to extract and transport changes in online transactional processing (OLTP) data systems to downstream systems such as data warehouses, data lakes and/or stream processors. In essence, CDC is implemented by writing to […]

6 days ago More
 

Kafka Connect mySQL Examples

In this Kafka Connect mysql tutorial, we’ll cover reading from mySQL to Kafka and reading from Kafka and writing to mySQL.   Let’s run this on your environment. Now, it’s just an example and we’re not going to debate operations concerns such as running in standalone or distributed mode.  The focus will be keeping it simple and get it working.  We can […]

26 days ago More
 

Spark Kinesis Example – Moving Beyond Word Count

If you are looking for Spark with Kinesis example, you are in the right place.  This Spark Kinesis tutorial intends to help you become better at integrating the two.  In this post, I’m going to provide a custom Spark Kinesis code example and a screencast of running it.  We’re going to cover running, configuring, sending […]

1 year ago 2 More
 

Spark Performance Monitoring Tools – A List of Options

Which spark performance monitoring tools are available to monitor the performance of your Spark cluster?  Let’s find out.  Before we address this question, I assume we already know Spark includes monitoring through the Spark UI.  In addition, Spark includes support for monitoring and performance debugging through the History Server as well as support for the […]

1 year ago 1 More
 

Spark FAIR Scheduler Example

Scheduling in Spark can be a confusing topic.  When someone says “scheduling” in spark, do they mean scheduling applications running on the same cluster?  Or, do they mean the internal scheduling of Spark tasks within the Spark application?  So, before we cover an example of utilizing the Spark FAIR Scheduler, let’s make sure we’re on the […]

1 year ago More
 

Spark Tutorial – Performance Monitoring with History Server

In this Apache Spark tutorial, we will explore the performance monitoring benefits when using the Spark History server.  This Spark tutorial will review a simple Spark application without history server and then revisit the same Spark app with the history server.  We will explore all the necessary steps to configure Spark History server for measuring […]

1 year ago More