This post is part 4 of a 4 part series:

Part 1 - Introduction
Part 2 - RabbitMq
Part 3 - ElasticSearch and Kibana
Part 4 - Tying it all together with LogStash

Introduction

In this article we will be setting up logstash so that it logs all messages from RabbitMq into ElasticSearch. We will also be setting up the indexes in Kibana so that we can start seeing the data.

If you have been following along you already have everything, if not then this is all you need.

Prequisites

  1. docker
  2. docker-compose
  3. shell window
  4. text editor
  5. docker-compose.yml
  6. docker-compose.override.yml

My setup

  1. Docker - Docker for windows (using linux containers)
  2. Docker-Compose - comes with docker for windows
  3. notepad++ or visual studio code
  4. powershell

If you are a windows user make sure that you set docker to run linux containers. There's not much in this domain that has an official windows image.

Since this is where all of the services will start to come together I will list out the whole compose file.

Note: white space is important in yml files
**Note: I am using version 3.7 of docker which requires docker-compose version 1.24. The error message when it tries to run it is pretty helpful and should tell you docker versions to run. If not then bump the version down until you reach a number that works.

Add logstash to docker-compose

version: "3.7"
services:
  rabbitmq:
    image: rabbitmq:management-alpine

  logstash:
    image: docker.elastic.co/logstash/logstash:7.1.1  
    depends_on:
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1

  kibana:
    image: docker.elastic.co/kibana/kibana:7.1.1
    depends_on:
      - elasticsearch

We have set up the other applications to match the default configuration so we didn't have to worry about any extra configuration. However you have to tell logstash how to handle the logs. So we will need to configure a pipeline. So we will copy the config and pipeline folders from logstash so that we have them. I always copy them into an easy to find directory and copy them over to where I want them so I always have a fallback copy. You could copy them directly in place, but I like having a reference point.

I am copying the entire directories because docker for windows doesn't have good support for copying individual files and from my understanding they intend to keep it that way.

To copy the files, in powershell run

docker cp <ContainerName>:/usr/share/logstash/config c:/logstash/config
docker cp <ContainerName>:/usr/share/logstash/pipeline c:/logstash/pipeline

copy those directories into the directory the docker-compose files are in. I structured mine like this

  • docker-compose.yml
  • docker-compose.override.yml
  • volumes
    • logstash
      • config
      • pipeline

now modify docker-compose.override.yml

version: "3.7"
services:
  rabbitmq:
    ports:
      - 15672:15672
  logstash:
     ports:
       - 9600:9600
     volumes:
        - ./volumes/logstash/config:/usr/share/logstash/config
        - ./volumes/logstash/pipeline:/usr/share/logstash/pipeline
  elasticsearch:
    environment:
      - discovery.type=single-node
    ports:
      - 9200:9200
      - 9300:9300
  kibana:
    ports:
      - 5601:5601

You don't have to expose any ports with logstash. Were exposing 9600 so that you can bring it up and see that it is running.

Volumes allow us to share a directory between our computer and the container so that either we can pass data into it or it can pass data back out. It is also how you allow containers to maintain state between restarts. Though we haven't really addressed that.

Alright lets do something a little more interesting.

If you look inside of the config folder at pipelines.yml it sets the directory for pipelines you can change it if you like but it's in a reasonably named folder already. Pipelines are what logstash uses to configure the processing of logs. All the files in the pipeline directory will be parsed and executed so make sure you only put pipeline conf files in here.

in the pipeline directory create a rabbitmq.conf file
populate it with

input {
  rabbitmq {
    id => "ApplicationRabbit"
    host => "rabbitmq"
    exchange => "app"
    exchange_type => "topic"
    key => "#"
    queue => "logs"
    durable => true
    user => "guest"
    password => "guest"
  }
}

filter {
  if [@metadata][rabbitmq_properties][timestamp] {
    date {
      match => ["[@metadata][rabbitmq_properties][timestamp]", "UNIX"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "rabbitmq-v1-%{+YYYY.MM.dd}"
    #user => "kibana"
    #password => "changeme"
  }
  stdout {
    codec => rubydebug
  }
}

**NOTE: the elasticsearch is expecting a valid json request complete with quotation marks. you will see elastic search errors in the logs with decent explanations if you send it something it doesn't like.

Input configure the location logstash gets the data. In our case RabbitMq. Here's a quick break down of the non-obvious parameters.

  1. exchange: tells logstash what exchange to bind to.
  2. exchange_type: allows logstash to create the exchange it it doesn't exist.
  3. queue: names the queue that logstash is bound, it will be created if it doesn't exist.
  4. key: this tells rabbitmq how to match messages that should go into this queue.
  5. durable: lets the queue and exchange survive if the client(logstash) disconnects.

Filter manipulates the data. This filter takes the timestamp from RabbitMq and sets it to the logstash metadata timestamp.There's a lot more that you can do with this but this is a good starter point.

output sets where the data should go, we are sending ours to elasticsearch and the console. The index is what you see inside of kibana. You should start with helpful name, it's possible to use other @metadata tags to build the name dynamically to set this as well but I thought it may be more understandable seeing it hardcoded. It should also end with a timestamp so you can build indexes of a specific day or time. The version piece in the middle is recommended so that you can version your logs incase they ever need to change.

if you set this up correctly you should be able to go to http://localhost:15672 and see an exchange called app and a queue called logs, sometimes this takes a minute because logstash seems to take the longest to start up.
If you publish any message to the exchange then logs will receive a copy.

Also worth noting is logstash and do a lot more than read messages from a message broker. You can configure it so that it gets a copy of the logs from your servers as well.

One last step after you send publish a message and it goes through to elasticsearch go to http://localhost:5601.

  1. Click on settings, it's the gear.
  2. Under elastic search click index management, you may already be there.
  3. You should see something an index called rabbitmq1-v1... if not then logstash
  4. Under kibana, click index patterns
  5. enter rabbitmq1-* it should turn to success so click next.
  6. set @timestamp and click the button
  7. Now go to the discover tab in the left menu, it's the compass
  8. You should see the messages that you have sent in.

Where to from here

In kibana, you can create visualizations and a dashboard so you can have a quick view into the health and status of your services. If your still manually entering messages this will probably be complicated at the moment, but maybe I'm just not creative enough.
The next step to achieve is to get your services to send messages to rabbitMq, so that they can communicate with each other and be logged. These systems will also needed to be hardened further if you plan to use them in a production environment.