Display your data in Kibana + Query it with the ES REST API

At this stage, our application is up and running, sending data to the ElasticSearch node in order to be indexed. Now it´s time for us to exploit it, so what we are going to do is:
- Start Kibana and define a simple date histogram
- Query Elastic Search REST API and SOAP UI
- Test aggregations a couple of simple aggregations on queries

Fire up Kibana

Unlike ElasticSearch, the Spring Cloud libraries do not seem to support the embedded execution of Kibana (I have opened a question in StackOverflow and a ticket in Spring GitHub project, I will update this post if any reply fixes the problem).
Therefore, if order to run Kibana, you will have to
- Download it from here.
- Run it by executing /bin/kibana.bat (or kibana.sh from a Unix system)

By default, Kibana Web GUI will be listening on port 5601 and it will try to connect to a ElasticSearch node at localhost:9200.

These parameters might be changed by editing the file /config/kibana.yml accordingly.

Register your index and fields

The first thing that Kibana will ask you is the name of the index you want to work with and whether it contains a temporal reference (it does)

• Fill the name of the index (stockquotations) and specify which field is the time reference (timestamp in this case)
• Go to Discover and select stock, timestamp and value from the available field list.
• You should see something like this image below (if not, expand the selected time frame to allow some result to be shown)

Now, let´s go to Visualizations and let´s create a date histogram with the evolution of our stocks along the time.

Stock date histogram

Kibana supports a large number of graph types, what we are showing here is just one of them, which is the most suitable for purpose in this case: The line chart.
Basically, two pieces of configuration need to be provided:

What is going to be shown in the Y-axis

• This is very easy, we will show the average value of the field value.

What is going to be shown in the X-axis

• In this case two aggregations are needed:
• A date aggregation: Add a date histogram of the field timestamp. If you choose an automatic granularity, it will adapt seamlessly whenever you change the date range of the graph (i.e. granularity should be different if you are showing five years of data or five hours).
• We will need one line per stock: Add a Sub-bucket, choose Split Lines and split by terms of the field stock.

And voilá! You have a graph showing all stocks, which changes automatically if you change the date range (try showing a couple of years and then switch to a couple of weeks).

Finally is also interesting to perform further filtering once the graph is defined. For instance you might want to see two stocks, let´s say MAP.MC and SAN.MC. In that case, just enter this query in the field above the graph:

stock in (map.mc, san.mc)

Use SOAP-UI to interact with ElasticSearch REST API

SOAP-UI is an incredibly useful tool for any developer working in integration projects, web services (SOAP, REST) or even coding Web interfaces. Among many other features it offers:

- Possibility of creating clients from WSDL, WADL contracts

- Perform load testing

- Request manual creation and play/replay (this is my favorite)

I would´t like to make this too long, but perhaps in the future I can write on how this tool eases the development and testing of this type of interfaces. In the example below, I have used the request builder to perform a POST on http://localhost:9200/stockquotations/search while comfortable editing the JSON payload in a text box.

Example of an aggregation on a search query

The Search API is huge and I am not entirely familiar with it. You can find extensive documentation here. Anyway, we can check a really simple example where we will query our index and retrieve:
- The list of stocks present in our system, grouping the rest of the query by stock.
- Ordered by the average stock value, descending.
- A number of useful statistical parameters per stock (standard deviation,max,min, etc.)

This is the request:

And this is the response (part of it for the sake of brevity):

Cool, isn´t? Later on, we will develop a service to retrieve this parameters and make further use of them.

Calculating Moving Averages

In the technical analysis of stock markets, there are a group of calculations that are specially useful: Moving Averages (either simple or exponential, weighted, etc.).
ElasticSearch will support this type of aggregation starting  from version 2.0.0 (current is 1.7.0). A really nice feature to have.
See more info here.

What is next?

Once we have played a little bit with Kibana and the REST interface, is time to make some queries from our Spring Boot application and make it available in our own REST interface to other applications or to our custom Web interface. We will see that in the next article.

As usual, you can find the source code in this Git Repository.