This post is part 3 of a 4 part series:

Part 1 - Introduction
Part 2 - RabbitMq
Part 3 - ElasticSearch and Kibana
Part 4 - Tying it all together with LogStash

Introduction

In this portion we are going to set up ElasticSearch and Kibana. ElasticSearch is a database that handles free text indexing and queries very quickly. Making it quite good for analytics and data science related tasks. Kibana is a data visualization platform that utilizes ElasticSearch data to display complex information in a simpler format.

Prerequisites

If you have been following along you already have everything if not then this is all you need.

  1. docker
  2. docker-compose
  3. shell window
  4. text editor
  5. docker-compose.yml
  6. docker-compose.override.yml

My setup

  1. Docker - Docker for windows (using linux containers)
  2. Docker-Compose - comes with docker for windows
  3. notepad++ or visual studio code
  4. powershell

If you are a windows user make sure that you set docker to run linux containers. There's not much in this domain that has an official windows image.

Note: white space is important in yml files
Open docker-compose.yml and add

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
kibana:
  image: docker.elastic.co/kibana/kibana:7.1.1

this tells docker what images to get and is what is used in docker-compose up is run.

Open docker-compose.override.yml and add

elasticsearch:
  environment:
    - discovery.type=single-node
  ports:
    - 9200:9200
kibana:
  ports:
    - 5601:5601

now run docker-compose up

environment variables allow us to pass configuration into a container to be used. In this case we are telling elastic search that there is only going to be one instance. This fixes and issue where it stops running in docker with the error:

Native controller process has stopped - no new native processes can be started

http://localhst:9200 is a rest api for elasticsearch. Comes in handy if you need to just see if it is running but can also be used to issue requests against the data.

http://localhst:5601 will take you to Kibana. I would recommend downloading some of the test data and looking at the different visualizations and things. If this is your first experience with data visualization tools then it is wicked awesome.

Commands to get config

ElasticSearch config files. The important one is elasticsearch.yml

docker cp <ContainerName>:/usr/share/elasticsearch/config C:/elasticsearch

Kibana config file

docker cp <ContainerName>:/usr/share/kibana/config C:/kibana