Spark Performance Monitoring Tools – A List of Options

Which spark performance monitoring tools are available to monitor the performance of your Spark cluster?  Let’s find out.  Before we address this question, I assume we already know Spark includes monitoring through the Spark UI.  In addition, Spark includes support for monitoring and performance debugging through the History Server as well as support for the […]

2 days ago More
 

Spark FAIR Scheduler Example

Scheduling in Spark can be a confusing topic.  When someone says “scheduling” in spark, do they mean scheduling applications running on the same cluster?  Or, do they mean the internal scheduling of Spark tasks within the Spark application?  So, before we cover an example of utilizing the Spark FAIR Scheduler, let’s make sure we’re on the […]

6 days ago More
 

Spark Tutorial – Performance Monitoring with History Server

In this Apache Spark tutorial, we will explore the performance monitoring benefits when using the Spark History server.  This Spark tutorial will review a simple Spark application without history server and then revisit the same Spark app with history server.  We will explore all the necessary steps to configure Spark History server for measuring performance […]

6 days ago More
 

Apache Spark Thrift Server Load Testing Example

Wondering how to do perform stress tests with Apache Spark Thrift Server?  This post will describe one way to do it. What is Apache Spark Thrift Server?   Apache Spark Thrift Server is based on the Apache HiveServer2 which was created to allow JDBC/ODBC clients to execute SQL queries using a Spark Cluster.  From my […]

9 days ago More
 

Spark Thrift Server with Cassandra Example

Want to use joins with Cassandra?  Or, help people already familiar with SQL leverage your Spark infrastructure?  They can use their existing SQL based tools they already know such as Tableau or even MS Excel.  Maybe both?  This post describes how to provide answers.  It describes how to configure the Apache Spark Thrift Server with […]

1 month ago More
 

Spark Streaming with Kafka Example

Spark Streaming with Kafka is becoming so common in data pipelines these days, it’s difficult to find one without the other.   This tutorial will present an example of streaming Kafka from Spark.  In this example, we’ll be feeding weather data into Kafka and then processing this data from Spark Streaming in Scala.  As the data […]

5 months ago More
 

Spark Command Line Arguments in Scala Example

The primary reason why we want to use Spark command line arguments is to avoid hard-coding values into our code. As we know, hard-coding should be avoided because it makes our application more rigid and less flexible. For example, let’s assume we want to run our Spark job in both test and production environments. Let’s […]

5 months ago More