Fast prototyping with Zeppelin, Spark & Scala

Summary

It´s been a while since I wrote anything here (again) but I don´t have much free time nowadays. Currently I´m taking some training on Scala and Spark, which, by the way, brings us here today.

A recipe for quick prototypes for Data Analysis: Scala + Spark + Zeppelin

If you remember well, I wrote some time ago about some personal learning projects I was working into, which basically picked stocks price information from the Web (using Spring Integration) and ran a couple of Spark analysis that were lately displayed in an AngularJS interface.

Nothing complicated at all, but rather verbose and time consuming to set up, specially if you just want to learn the subject.

With Apache Zeppelin, Scala and Spark, a prototype with basically the same functionality, would be reduced to these two code snippets in a Zeppelin notebook:


That reads the data, creates a Dataframe with the price and the moving average and declares a temporal table.
Later on, you can query it with a regular SQL query and its result set will be available as data source for a number of graph types:


This will produce this nice looking graph after just some minutes of coding, showing the index price and our friend, the Simple Moving Average (SMA):

The notebook I´m talking about is hosted in ZeppelinHub and you can access it here.

And that's it! No project setup, no unnecessary pain with Angular JS libraries and no horrible Java Spark API. If you are interested, you can find below the links to the articles explaining the topics seen here today and how were they implemented in the traditional way.

Cheers!

Resources

- Consuming a REST service in Scala
- Calculating Moving Averages with Spark MLLib
- Stock quotations as input for your Spring application
- Calculating and stock indicators with AngularJS and Spark