[Code Conference: Big Data 2016] Elastic Search + LogStash + Kibana

This is great talk about ELK stack (Elastic Search + Logstash + Kibana) by K.Supaket, software engineer at Agoda. He is working on big data log analysis which proved to be a challenging task.

Problems in log analysis

Non-consistent log format from multiple systems e.g. IIS, Tomcat, Apache, Android, iOS
Logs from each system are collected separately (decentralized) which is difficult to search.
Expert knowledge requirement e.g. real-time logging

What is Elastic Search

Schema-free database (JSON based) with REST. Distributed system (manual or auto) and scalable. No configuration required to start.

And the best part.. it’s FREE (Apache license 2.0)

Elastic Search Terminology

Index – Indexing data to improve reading speed.
Replication – Replicate data across different machines, allowing scalability in reading.
Sharding – Split logical data across different machines, allowing scalability in writing.

Use cases of Elastic Search

Product search engine
Product search results weight

What is Logstash

Manage events and logs to collect data, filter data, and output data. It helps centralized logs from every system.

Searchable logs and can make a report from logs.

Accept input and output to many popular libraries to collect data e.g. elasticsearch, log4j, kafka etc.

What is Kibana

Kibana is website platform on top of Elastic Search, which can query and visualize data from Elastic Search.

ELK Stack

Getting Started

We can try ELK from docker e.g. https://github.com/deviantony/docker-elk

There was a demo in video during 21:00 minutes (in Thai)

References

Original video can be watched here (Thai): https://www.youtube.com/watch?v=jONJ1f-1InE