Spring Boot/Integration gateway microservice

Summary

As we saw in the previous article, we created a stream of information coming from stock quotations that we will process in our application. What are the objectives? For the moment two:

• De-serialize the information and split it in tuples of (Stock,Price,Date)
• Publish these items to two streams
• A simple Log
• To the ELK stack (Elastic Search + Logstash + Kibana). We won´t use Logstash, though.

As usual with Spring Boot + Integration, the implementation of this kind of services is extremely easy.

Creation of the application  with Spring STS

Of course you can do this manually or by using a Maven archetype. However, Spring STS provides a handy wizard for creating applications with Spring-Boot that apart from creating the basic project framework, it allows you to add whatever Spring modules you may consider needed:

1. Modules chosen:
• Spring-Integration: Support for integration flows and usage of reusable components (file reading, messaging and tons more...)
•  Spring-Elastic Search (Cloud): This includes, among other features, a ES client that facilitates the sending of information towards ES. Moreover, using the default configuration, it launches a local ES node to play with.
2. I manually added two dependencies:
• Spring-Integration-File: This will allow us to load files with historical data
• Elastic Search: This is needed to work with the latest Kibana version (This will be explained more in depth in the next article)

For the sake of clarity, you can see how the pom.xml looks like here.

Implementation of the Integration business logic

All is accomplished by means of an XML defining the flow, a couple of Java classes and an import in the main class. This can be reduced even more by using more auto-configuration features.

• Enable the Spring Integration support for you XML file containing the flow

• Write the flow, in a file called /META-INF/feedFlow.xml, which will contain:

1. A http-outbound-gateway, that polls regularly the URL where the stock quotations are published
2. A file reader that reads the local files containing historical data.
3. A splitter, that creates one "item" per stock quotation.
4. A logger that logs each item
5. A service activator that sends the information to Elastic Search through REST

Let´s take a look to the most relevant elements with some more detail:

HTTP part of the flow

1. A int-http:outbound-gateway performs a HTTP request to the specified URL. The result of the request (the whole CSV file) is sent to the quotations channel
2. The request is triggered after an int:inbound-channel-adapter, which is driven by a poller, injects a message in the trigger channel.

File part of the flow

It might be useful, for further studies to have historical quotations. In order to process them a int-file:inbound-channel-adapter is polling a local folder where the user can place the files that he/she would like to load. These files are read and send to the quotations channel.

In this really simple example, we can see the powerful abstraction that Spring Integration gives, as we can inject quotations independently from where they come, as long as they respect the appropriate contract (format).

Final part of the flow

Finally, the CSV is split  in lines by a custom implementation of a Splitter and the result is made public in a publisher-subscriber channel, which is the Spring equivalent to a JMS topic.

Two consumers feed on these messages:

- A really simple logger, that logs the whole message (headers included)

- A service activator that send the quotations to the Elastic Search node.

We will see, how do we publish the data to ES and in which format in the next article. Moreover, we will create a couple of nice graphs using the data indexed in ES.

Source code

You can find the code in this GitHub repository.

I´m still improving things, like the error handling, configuration and proper conversion from CSV to the data expected by ES. In any case, feel free to share your critics/feedback.