Thursday 19 November 2015

Dockercon 2015

Just returned from the DockerCon 15 that took place in Barcelona, Spain. It has been a somewhat tiring but interesting experience. Here there are some quick thoughts I´d like to share with you.

Brand new Docker functionalities


In the general session, at the very beginning of the conference, some new features were show-cased and seemed  very interesting:
  • Production-ready Docker Swarm (which handles the creation and orchestration of a large number of containers in distributed applications).
  • Out of the box multi-host networking.
These two features were demonstrated by creating 50.000 containers on 1.000 Amazon Web Services nodes. However, I would have loved to see them actually performing some work, (rather just starting up). Specially hard would have been to share some workload and perform some kind of coordination...

Here you can see the video from the general session:


The demo done on the Docker Toolbox was quite interesting, it might be a really good tool in order to ramp up new developers coming to your project.

Tuesday 27 October 2015

Introduction to CI and Cloud deployment with GitHub,Travis and Heroku

Introduction


These past weeks, while (eventually) working in my Stokker project, I noted the lack of continuous build that I usually enjoy at work. I committed many bugs to my code-base and even, non-compiling code!

So, what are the alternatives to fix this situation? Perhaps having a local Jenkins environment? Too hard to maintain. No CI at all? Not feasible if we pretend that somebody might cooperate in the future.

One possible solution: Use Github integration with Travis CI and Heroku


This is one of many and, to be fair, I did´t do a thorough study on the different choices before trying it. I noticed that JHipster is using it as CI system for part of its code-base, though.

Basically, what Travis-CI does is to poll regularly your repository and perform a build based on the parameters specified in a file called .travis.yml.

Here, we can see a very basic example on how does this file look like:
Basically, we are stating that:
  • This is a Java project. As no build.gradle file has been provided, it will execute these Maven commands:
    • mvn clean install
    • mvn test
  • We intend to deploy the result of the successful build to Heroku
    • You can get the API Key later on, after registering in Heroku.

Hint on working with the .travis.yml file: One handy tool is this YAML validator, specially due to the cryptic error messages given by Travis-CI.

Once a change in the master branch is detected, Travis-CI will pull the code and perform a Maven build. If a compilation or test failure is detected the process will be stopped and notification mails will be sent.

This is how a successful build looks like:


As said, once you have a successful build, Travis CI will trigger a Heroku deployment. By the way, what is Heroku? Heroku is a commercial PaaS that also offers some free functionality for hobby programmers and people learning. If you want to test it, follow this steps:

  • Register in Heroku with your personal details.
  • Create an application and link it with your Github:
    • By doing so, you will be provided with an application key (which was inserted in the YML snippet above).
  • Choose the "Free" dyno mode. This mode offers you very limited resources and the machine is shut down after 60 minutes of inactivity.
    • By the way, these are the different plans available:



The way in which your application is executed, is defined in a Procfile (this file should be placed in the same folder of .travis.yml)



Running example


You can check the unfinished application running here: https://stokker.herokuapp.com

Update on November 1st 2015: I had to separate Stokker and Portfolio Manager into two different Github repositories (and two Heroku applications) in order to be able to run both of them so the URL for the Portfolio manager in Heroku is now: https://stokker-portfolio-manager.herokuapp.com

  • Please excuse me for the ugly interface, learning AngularJS is still one of my non achieved goals!). Moreover I have not yet provided a CRUD JavaScript interface for the entities managed in the project (Portfolio, Market Position and Stock). If you want to interact with the database, you can use the Spring Data REST endpoints:
  • You will notice that no price is used for the portfolios, as I still have not figured out how to run more that one application per repository.
  • Kibana functionality won´t be available.


Manage your application with Heroku toolbelt

Heroku offers you a tool called Heroku toolbelt that you can install in your local computer and use it to:
  • Manage and monitor your applications:
    • You can login and check the application logs.
    • You can deploy directly by performing a git push heroku.
  • Deploy your applications in a local environment in order to ensure that everything is OK before pushing your code.

Resources

This is a really shallow introduction to Travis CI and Heroke. I am astonished, as the more I read the more documentation I found. Here you might find some interesting links in case you want to continue learning on this topic:


Hope that you find this quick introduction interesting, any feedback is always more than welcome! You can find the code of the unfinished application here.

Wednesday 14 October 2015

Integrate your Kibana dashboard in your Spring Boot + AngularJS application

Summary


Let´s review the process to follow in order to:

  •  Provide your Spring Boot application with a very basic AngularJS interface
  •  Show an already defined Kibana dashboard and interact with it.

The application for which we are going to define the user interface is called Portfolio Manager, a really simple Spring Boot application with some JPA entities (stock, position, portfolio, etc.) exposed via REST.
In order to feed the Elastic Search node from which Kibana will read, we will use Stokker (application described in past articles).

See this high-level diagram for a clearer picture of the system:



Both of them can be checked in this Git Hub repository along with the instructions to run all the components.

Wednesday 30 September 2015

Summer rentrée + upcoming DockerCon Europe 2015


Sorry, it´s being a while since my last post but it has been a terrible time for writing (summer holidays, good weather, "la feria de Málaga", ...).
I have some unfinished topics that I´m working with, and I would like to polish them before writing here, so stay tuned:

  • Spring Boot 1.3.0 + Elastic Search 2.0 together (I´m having problems trying to run them together, beta components are packed with bugs pending to be fixed in the incoming releases).
  • Setting up OAuth2 in my micro-services environment along with the rest of Netflix components (Eureka, Zuul, ...).
  • Exploring on the usage of Spring Data Rest to make it extremely easy to provide your persisted entities a nice REST interface.
  • ...

In the meantime, something new and exciting has come up: Some colleagues and myself will attend the DockerCon Europe 2015 in Barcelona!

The agenda is still open, but I have watched some of the videos of the recent SFO DockerCon 2015 and some of them are pretty interesting, specially when it comes to dealing with services deployed in a Cloud environment that you have to deploy/undeploy, orchestrate and monitor.

I really hope that this event meets my expectations and that I can come back and share with you plenty of interesting features worth learning.

Here you have some interesting resources from Youtube regarding the SFO DockerCon:

Specially interesting, as they are really close to what I´m doing, are these two videos:
Finally, as I am somewhat a novice in this technology, I am following this official tutorial series (given by one of the SFO speakers) which is a really nice place to start learning:

That´s all folks (for now).

Wednesday 12 August 2015

Historical Stock Quotations as input for your Spring Application

Update on July 2nd 2017

Unfortunately, the API described in this article has been changed and now it requires a session cookie to access the data in CSV format. I guess the guys on Yahoo do not like to display their data without having the chance of displaying their ads...
Still, you might download the data manually (example URL here).
In the meantime, I´m looking for an alternate source of prices, I´ll keep you posted.

Summary


In a previous article, we defined how to get new stock quotations as soon as they were published in Google Finance (of course, after a 20-minute delay, this is a free service after all).
However, it was hard to study this data without having any historical data to compare with, so we defined a manual process to load some historical prices.

This manual process is very inefficient, though. You need to do it manually and there is a data gap between the last file update and the current quotations.

Solution: Yahoo finance historical quotations


We know that this service is not suitable for "real-time" quotations in case of the Spanish stock exchange, however, it is perfectly good for getting historical quotations during a period of time and offering more information such as:
- Volume
- Open, close, high and low prices
- Statistical data, such as mobile means, etc.

API Description


There seems to be no documentation for this interface at all (or at least, I have not been able to find it), however, I found some information about it buried in this 2009 blog article.

Basically, you need to compose a URL with the stock ticker and some time delimiters, for instance:


New Spring integration flow


As I had to replace the logic for reading the old static files, I created a new flow defined entirely in a separated XML. Basically what it does is:
- Reads from the application settings which stocks are we going to retrieve.
- Reads the time range to recover.
- Performs one request per Stock to the specific URL.
- Parses and converts the received CSV to our internal format.
- Publishes the new POJOs in to the topic so the  consumer can use this information seamlessly (it does not really care how the POJOs were built as long as the contract is respected).

Application.properties


HistoricalCSVRequestCreator

This class will receive a signal from Spring ApplicationContext as soon it is initialised (as it implements the interface ApplicationListener), it will parse the property containing the stocks to be retrieved and finally it will inject one message in the downstream channel per stock (Note that the ticker code is set as message header).


HTTP Outbound Gateway

This is the XML configuration for the component that will perform the request. Note how the target ticket is retrieved from the message header ticker and how the range of time is taken from the properties file.
And that´s it! As soon as you see the application running, a stream of data will be loaded into the ElasticSearch node so you can perform studies with wider statistical information.

As usual you can find the code in this GitHub repository. Feedback is always welcome!

Related articles


  1. Live Stock data as input for your Spring application
  2. Implement a Spring Integration flow with Spring Boot
  3. Integrate the application with ElasticSearch
  4. Represent and search your data with Kibana
  5. Deploy your Spring Boot microservice in a Docker container


Sunday 9 August 2015

Deploy your Spring Boot microservice + Kibana in a Docker container

Summary


In this step, we are are going to deploy our solution to a Docker container. This eases deployment as we don´t have to start/stop one by one each module (ie. the microservice, ElasticSearch and Kibana) and the port configuration is much simpler to do.

Docker installation


If you are running on a Linux distribution, Docker comes out of the box. In Windows or MacOS, it´s a bit more complicated. One option to run Docker on those systems is to use Boot2Docker, (which actually runs a virtualized Linux with Docker installed in it).
You can find the installation instructions here.

Creation of the Docker image


The specifications of the Docker Image are defined in a DockerFile file, placed within the application resources. Here is an explanation on each line of the file:


Note: I based this file on the one used in this example.

In order to create the Docker image to run, you need to get access to the DockerFile file and our application´s executable Jar file generated by the Maven build. You can achieve this in two steps:

  • Create a shared folder called "stokker" in your VirtualBox agent. This folder should point to the folder in your local host where the code is compiled.
  • Run this command to mount that folder in the embedded Linux OS running Docker:
    • sudo mount -t vboxsf stokker /mnt/stokker

Now, change the current directory to /mnt/stokker/target and run this command to register locally the image:

docker build -t victor-ferrer/stokker


Run it!


First, find out the image id of the newly created image:
docker images

By executing this command you will launch the image execution:
docker run -i <image_id> -P

Note: The -P option enables the port exposure defined in the DockerFile. If that parameter is missing, you will have to manually provide the mappings (along with the -p option).

Note #2: If you are running the Docker image in VirtualBox, you will have to expose the same ports again, so they can be used from outside VirtualBox. In order to do so, open the VM network configuration and add the following port forwarding:





That should be all. You can check any of the Spring Boot Actuation endpoints exposed in port 8080 or the Elastic search cluster status endpoint:

  • http://<container ip address>:8080/health
  • http://<container ip address>:9200/_cluster/health

You can get the <container ip address> by executing a netstat command and choosing the IP associated to the VirtualBox container.

Note: I´m still working on launching Kibana along our application and ElasticSearch...
Moreover, it is stated in the Docker best practices, that is not a good idea to run several processes in the same Docker container, as that impairs horizontal scalability and application decoupling. So, I´m not sure whether I will finally add Kibana to this image or to a new one...

Related articles


  1. Define the application input: Google Spreadsheets
  2. Implement a Spring Integration flow with Spring Boot
  3. Integrate the application with ElasticSearch
  4. Represent and search your data with Kibana

Friday 31 July 2015

Display your data in Kibana + Query it with the ES REST API


At this stage, our application is up and running, sending data to the ElasticSearch node in order to be indexed. Now it´s time for us to exploit it, so what we are going to do is:
- Start Kibana and define a simple date histogram
- Query Elastic Search REST API and SOAP UI
- Test aggregations a couple of simple aggregations on queries

Fire up Kibana

Unlike ElasticSearch, the Spring Cloud libraries do not seem to support the embedded execution of Kibana (I have opened a question in StackOverflow and a ticket in Spring GitHub project, I will update this post if any reply fixes the problem).
Therefore, if order to run Kibana, you will have to
- Download it from here.
- Run it by executing /bin/kibana.bat (or kibana.sh from a Unix system)

By default, Kibana Web GUI will be listening on port 5601 and it will try to connect to a ElasticSearch node at localhost:9200.

These parameters might be changed by editing the file /config/kibana.yml accordingly.

Register your index and fields


The first thing that Kibana will ask you is the name of the index you want to work with and whether it contains a temporal reference (it does)
  • Fill the name of the index (stockquotations) and specify which field is the time reference (timestamp in this case)
  • Go to Discover and select stock, timestamp and value from the available field list.
    • You should see something like this image below (if not, expand the selected time frame to allow some result to be shown)
Now, let´s go to Visualizations and let´s create a date histogram with the evolution of our stocks along the time.

Stock date histogram

Kibana supports a large number of graph types, what we are showing here is just one of them, which is the most suitable for purpose in this case: The line chart.
Basically, two pieces of configuration need to be provided:

What is going to be shown in the Y-axis

  • This is very easy, we will show the average value of the field value.

What is going to be shown in the X-axis

  • In this case two aggregations are needed:
    • A date aggregation: Add a date histogram of the field timestamp. If you choose an automatic granularity, it will adapt seamlessly whenever you change the date range of the graph (i.e. granularity should be different if you are showing five years of data or five hours).
    • We will need one line per stock: Add a Sub-bucket, choose Split Lines and split by terms of the field stock.
And voilá! You have a graph showing all stocks, which changes automatically if you change the date range (try showing a couple of years and then switch to a couple of weeks).

Finally is also interesting to perform further filtering once the graph is defined. For instance you might want to see two stocks, let´s say MAP.MC and SAN.MC. In that case, just enter this query in the field above the graph:

stock in (map.mc, san.mc)


Use SOAP-UI to interact with ElasticSearch REST API

SOAP-UI is an incredibly useful tool for any developer working in integration projects, web services (SOAP, REST) or even coding Web interfaces. Among many other features it offers:
- Possibility of creating clients from WSDL, WADL contracts
- Perform load testing
- Request manual creation and play/replay (this is my favorite)

I would´t like to make this too long, but perhaps in the future I can write on how this tool eases the development and testing of this type of interfaces. In the example below, I have used the request builder to perform a POST on http://localhost:9200/stockquotations/search while comfortable editing the JSON payload in a text box.

Example of an aggregation on a search query

The Search API is huge and I am not entirely familiar with it. You can find extensive documentation here. Anyway, we can check a really simple example where we will query our index and retrieve:
- The list of stocks present in our system, grouping the rest of the query by stock.
- Ordered by the average stock value, descending.
- A number of useful statistical parameters per stock (standard deviation,max,min, etc.)

This is the request:

And this is the response (part of it for the sake of brevity):


Cool, isn´t? Later on, we will develop a service to retrieve this parameters and make further use of them.


Calculating Moving Averages

In the technical analysis of stock markets, there are a group of calculations that are specially useful: Moving Averages (either simple or exponential, weighted, etc.).
ElasticSearch will support this type of aggregation starting  from version 2.0.0 (current is 1.7.0). A really nice feature to have.
See more info here.

What is next?

Once we have played a little bit with Kibana and the REST interface, is time to make some queries from our Spring Boot application and make it available in our own REST interface to other applications or to our custom Web interface. We will see that in the next article.

As usual, you can find the source code in this Git Repository.


Sunday 26 July 2015

Integrate ElasticSearch with Spring-Boot

Summary

As commented in the previous post, the destinations for the recovered information will be:
- A simple log
- An Elasticsearch node, fur further data indexing and analysis.
How do we accomplish this? Very easy.

As we saw, during the project creation, we included the dependency spring-boot-starter-data-elasticsearch. This dependency will provide us with:
- A "template" API to perform  REST operations in towards an ES node, but without all the REST boilerplate.
- An embedded ES node, that works great in order to perform tests and help you understand ES.

Launching an ES node along with your Spring-Boot application:

If you want to launch an embedded ES node, you just need to:

  • Enable it in your application .properties file by setting spring.data.elasticsearch.properties.http.enabled=true.
  • Check that is properly running this curl command:
    • $ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
  • Some cluster health statistics will be displayed (name, shards available, etc.)

Sending the information to ES

If you recall, our Spring Integration flow ended in a Service Activator. This component specified that @ELKClient.pushToELK()  would be invoked along with the payload of the message received (that is the CSV quotation). Well, let´s see the code used to perform the sending (Which I think is pretty much self-explanatory)

Building the document to be indexed

Finally we just need to create a simple bean, with some annotations specifying which fields are to be sent, analyzed and stored and in which format:

Next Steps

Once we run the application, it will load some files containing historical data and it will query the spreadsheets stored in Google Docs. Each entry will be converted to a Java Object and sent to Elastic Search. In the next post we will see how can we build some nice graphs using Kibana (part of ELK stack) and the data we have indexed. One example (showing the historical data):

Example of one dashboard created with Kibana and the historical data

Moreover, we will see how can we deploy the micro-service, the elastic search node and Kibana to the cloud, so we can continuously gather stock quotations and perform better analysis.

Source code

You can find the code in this GitHub repository.

Friday 24 July 2015

Spring Boot/Integration gateway microservice


Summary


As we saw in the previous article, we created a stream of information coming from stock quotations that we will process in our application. What are the objectives? For the moment two:
  • De-serialize the information and split it in tuples of (Stock,Price,Date)
  • Publish these items to two streams
    • A simple Log
    • To the ELK stack (Elastic Search + Logstash + Kibana). We won´t use Logstash, though.

As usual with Spring Boot + Integration, the implementation of this kind of services is extremely easy.

Creation of the application  with Spring STS

Of course you can do this manually or by using a Maven archetype. However, Spring STS provides a handy wizard for creating applications with Spring-Boot that apart from creating the basic project framework, it allows you to add whatever Spring modules you may consider needed:
  1. Modules chosen:
    • Spring-Integration: Support for integration flows and usage of reusable components (file reading, messaging and tons more...)
    •  Spring-Elastic Search (Cloud): This includes, among other features, a ES client that facilitates the sending of information towards ES. Moreover, using the default configuration, it launches a local ES node to play with.
  2. I manually added two dependencies:
    • Spring-Integration-File: This will allow us to load files with historical data
    • Elastic Search: This is needed to work with the latest Kibana version (This will be explained more in depth in the next article)
For the sake of clarity, you can see how the pom.xml looks like here.

Implementation of the Integration business logic

All is accomplished by means of an XML defining the flow, a couple of Java classes and an import in the main class. This can be reduced even more by using more auto-configuration features.

  • Enable the Spring Integration support for you XML file containing the flow


  • Write the flow, in a file called /META-INF/feedFlow.xml, which will contain:
    1. A http-outbound-gateway, that polls regularly the URL where the stock quotations are published
    2. A file reader that reads the local files containing historical data.
    3. A splitter, that creates one "item" per stock quotation.
    4. A logger that logs each item
    5. A service activator that sends the information to Elastic Search through REST
STS-generated diagram with the flow


Let´s take a look to the most relevant elements with some more detail:


HTTP part of the flow
  1. int-http:outbound-gateway performs a HTTP request to the specified URL. The result of the request (the whole CSV file) is sent to the quotations channel
  2. The request is triggered after an int:inbound-channel-adapter, which is driven by a poller, injects a message in the trigger channel.
File part of the flow
It might be useful, for further studies to have historical quotations. In order to process them a int-file:inbound-channel-adapter is polling a local folder where the user can place the files that he/she would like to load. These files are read and send to the quotations channel.

In this really simple example, we can see the powerful abstraction that Spring Integration gives, as we can inject quotations independently from where they come, as long as they respect the appropriate contract (format).

Final part of the flow
Finally, the CSV is split  in lines by a custom implementation of a Splitter and the result is made public in a publisher-subscriber channel, which is the Spring equivalent to a JMS topic.
Two consumers feed on these messages:
- A really simple logger, that logs the whole message (headers included)
- A service activator that send the quotations to the Elastic Search node.

We will see, how do we publish the data to ES and in which format in the next article. Moreover, we will create a couple of nice graphs using the data indexed in ES.

Source code

You can find the code in this GitHub repository.

I´m still improving things, like the error handling, configuration and proper conversion from CSV to the data expected by ES. In any case, feel free to share your critics/feedback.

Thursday 23 July 2015

Stock quotations as input for your Spring Integration App

Summary


As you might know Google deprecated its Google Finance API some time ago and other options such as Yahoo Finance API do not seem to work when dealing with Stocks outside the US (i.e. Spanish stocks are not supported).

So here I found an option of getting Stock prices by means of using a Google Spreadsheet as a source of information. Later on, we will consume this source of information with Spring Integration.

Creating and publishing your Google Spreadsheet containing stock quotations

If you are familiar with Google Docs and Spreadsheets, you will know for sure how simple is to create such documents, so I will omit the creation of the spreadsheet for the sake of clarity.

We will use the function GOOGLEFINANCE to retrieve the prices and timestamp of a number of stocks (there are many more features, you can see full function documentation is Google Spreadsheets) using three columns:

  • Column A: Stock ticker
  • Column B: Price
    • The ticket will be taken from column A
    • =GOOGLEFINANCE($A1)
  • Column C:
    • Last trade dateThe ticket will be taken from column A
    • =GOOGLEFINANCE($A1;"tradetime")
With this setup, you will be able to write in column A as much stocks as you want and extend the formula to create a table. See this example (all but one are spanish stocks, OHI trades in the NYSE, note that for today we still don´t have a quotation).



Publish your sheet

Just go to File... Publish in the Web... and choose which sheets you want to publish and CSV format as output (of the formats available is the easiest by far to read later on by our application).
Check out this example here.

Next steps

After this simple set up, you will have a simple source of information with the stock parameters we need. In the next article, we will start building our Spring application (using Boot and Integration) to consume this source of data, store and analyze it.
Bear in mind that this free stock data is not suitable for automatic trading as it is not 100% reliable and usually come with a delay of around 20 minutes.


Update


This article refers to getting "live" stock data (i.e. delayed by 20 minutes). If you are also interested in getting historical stock data, Yahoo finance is still a good choice for you.  See this other article with more information on how to query the API provided by Yahoo Finance.


Hello World!


Yet another software blog. I know, that the idea is, by far, not the most original in the world.
However, in the past weeks, I found myself involved in a series of learning tasks that I have the feeling will be more useful for me and others if made public and accessible for everyone instead of being written to a .txt file and the source code stored in a local sandbox (only to be lost in the next HDD failure).

Let´s see how it works...