Spark Thrift Server with Cassandra Example

Want to use joins with Cassandra?  Or, help people already familiar with SQL leverage your Spark infrastructure?  They can use their existing SQL based tools they already know such as Tableau or even MS Excel.  Maybe both?  This post describes how to provide answers.  It describes how to configure the Apache Spark Thrift Server with […]

17 days ago More
 

Spark Streaming with Kafka Example

Spark Streaming with Kafka is becoming so common in data pipelines these days, it’s difficult to find one without the other.   This tutorial will present an example of streaming Kafka from Spark.  In this example, we’ll be feeding weather data into Kafka and then processing this data from Spark Streaming in Scala.  As the data […]

4 months ago More
 

Spark Command Line Arguments in Scala Example

The primary reason why we want to use Spark command line arguments is to avoid hard-coding values into our code. As we know, hard-coding should be avoided because it makes our application more rigid and less flexible. For example, let’s assume we want to run our Spark job in both test and production environments. Let’s […]

4 months ago More
 

Spark Performance Monitoring with Metrics, Graphite and Grafana

Spark is distributed with the Metrics Java library which can greatly enhance your abilities to diagnose issues with your Spark jobs.  In this post, we’ll cover how to configure Metrics to report to a Graphite backend and view the results with Grafana. Optional, 20 Second Background If you already know about Metrics, Graphite and Grafana, […]

9 months ago More
 

Spark Broadcast and Accumulator Examples in Scala

Spark Broadcast and Accumulator Overview So far, we’ve learned about distributing processing tasks across a Spark cluster.  But, let’s go a bit deeper in a couple of approaches you may need when designing distributed tasks.  I’d like to start with a question.  What do we do when we need each Spark worker task to coordinate certain […]

1 year ago More
 

Apache Kafka – A Few Things You Should Know

What is Apache Kafka? Simply put, Kafka is a messaging system, making it possible for data to be moved in between systems. For the laymen, however, the concept may not be easy to understand. Technically, it is defined as a commit log service that is distributed, partitioned, and replicated. It provides a messaging system that […]

1 year ago More
 

IntelliJ Scala and Apache Spark – Well, Now You Know

IntelliJ Scala and Spark Setup Overview In this post, we’re going to review one way to setup IntelliJ for Scala and Spark development.  The IntelliJ Scala combination is the best, free setup for Scala and Spark development.  And I have nothing against ScalaIDE (Eclipse for Scala) or using editors such as Sublime.  I switched from […]

1 year ago 3 More