Install ELK stack with Redis

In this post we will install the ELK stack (Elasticsearch, Logstash and Kibana) coupled with Redis.

The ELK stack consists of :

  • Elasticsearch : It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents
  • Logstash : It is a tool for managing events and logs
  • Kibana : Is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data

In this article i will install ELK stack with a Redis server to collect and analyze AWS ELB logs, the Redis server used here just to show you how you can cache you data in case of real time data analysis.

Our architecture is composed of 5 servers and an S3 bucket that holds our ELB logs :

  • Shipper : in this server we install logstash server to get all logs from the S3 bucket
  • Redis : in this server we install Redis cache to cache all logs from the shipper
  • Indexer : in this server we install logstash to get all logs from Redis server and index them and send all data to ElasticSearch server
  • Elasticsearch : in this server we install ElasticSearch
  • Kibana : in this server we install Kibana

All servers used in this article are Debian Jessie servers, the picture below show the data flow.

 

  • Installing Redis server :

Then modify the Redis conf file /etc/redis.conf by setting daemonize to yes :

Run and check if redis server is OK :

  • Installing Shipper server :

Installing Oracle java :

Installing Logstash :
Create aws authentication file for S3 bucket /opt/logstash/aws_credentials.yaml with this contents :
Create the shipper conf file /etc/logstash/conf.d/shipper.conf with this contents :
 Start logstash shipper :
  • Installing Indxer server :

Installing Oracle java :

Installing Logstash :

Create the indexer conf files /etc/logstash/conf.d/input-file.conf with this contents :

And /etc/logstash/conf.d/output-stdout.conf with this contents :
Start logstash indexer :

  • Installing ElasticSearch server :

Installing Oracle java :

Installing ElasticSearch :

Append this lines to /etc/elasticsearch/elasticsearch.yml :
And append this line /etc/default/elasticsearch :
Start ElasticSearch :
You can check that you got some logs by listing all documents :
  • Installing Kibana server :
Installing apache :
Creating kibana vhost file /etc/apache2/sites-available/kibana.conf :
Choose a password and access your Kibana server with user : kibana and password : choose one
Start kibana :
Enable apache kibana vhost and restart apache :