Spark Streaming Testing with Scala Example

Spark Streaming Testing
/ Categories: Spark Comments: no comments

Spark Streaming Testing How do you create and automate tests of Spark Streaming applications?  In this post, we’ll show an example of one way in Scala.  This post is heavy on code examples and has the added bonus of using a code coverage plugin. Are the tests in this tutorial examples unit tests?  Or, are

read more

Spark Streaming Example – How to Stream from Slack

Spark Streaming Example
/ Categories: Spark Comments: 3 Comments

Let’s write a Spark Streaming example in Scala, which streams from Slack.  This post will show how to write, configure and execute the code, first.  Then, the source code will be examined in detail.  If you don’t have a Slack team,  you can set one up for free.   We’ll cover that too. Let’s start

read more

How-To Apache Spark Streaming with Scala Part 1

Spark Streaming with Scala
/ Categories: Spark Comments: no comments

Let’s start Apache Spark Streaming by building up our confidence with small steps.  These small steps will create the forward momentum needed when learning new skills.  The quickest way to gain confidence and momentum in learning new software development skills is executing code that performs without error. In this post, we’re going to setup and

read more

Spark Streaming from Learning Spark Chapter 10

Spark Streaming
/ Categories: Spark, Summary Series Comments: no comments

Spark Streaming Spark Streaming based applications are tracking statistics about page views in real time, train a machine learning model, or automatically detect anomalies. The abstraction in Spark Streaming is called DStreams or discretized streams. A DStream is a sequence of data which arrives over time. Internally, each DStream is represented as a sequence of

read more