This is great talk about ELK stack (Elastic Search + Logstash + Kibana) by K.Supaket, software engineer at Agoda. He is working on big data log analysis which proved to be a challenging task.
Problems in log analysis
- Non-consistent log format from multiple systems e.g. IIS, Tomcat, Apache, Android, iOS
- Logs from each system are collected separately (decentralized) which is difficult to search.
- Expert knowledge requirement e.g. real-time logging
What is Elastic Search
Schema-free database (JSON based) with REST. Distributed system (manual or auto) and scalable. No configuration required to start.
And the best part.. it’s FREE (Apache license 2.0)
Elastic Search Terminology
- Index – Indexing data to improve reading speed.
- Replication – Replicate data across different machines, allowing scalability in reading.
- Sharding – Split logical data across different machines, allowing scalability in writing.
Use cases of Elastic Search
- Product search engine
- Product search results weight
What is Logstash
Manage events and logs to collect data, filter data, and output data. It helps centralized logs from every system.
Searchable logs and can make a report from logs.
Accept input and output to many popular libraries to collect data e.g. elasticsearch, log4j, kafka etc.
What is Kibana
Kibana is website platform on top of Elastic Search, which can query and visualize data from Elastic Search.
We can try ELK from docker e.g. https://github.com/deviantony/docker-elk
There was a demo in video during 21:00 minutes (in Thai)
Original video can be watched here (Thai): https://www.youtube.com/watch?v=jONJ1f-1InE