ELK Stack Playground

What is ELK?

ELK stack is an acronym that consists of 3 different projects: Elasticsearch, Logstash, and Kibana. These tools combined let you collect logs from different sources, then analyze and visualize this data. So they help with monitoring, troubleshooting, analyzing security incidents, keeping an eye on resource usage, and so on.

Elasticsearch: It is a distributed search engine. It is built on Apache Lucene - an open-source search engine library. Elasticsearch can store multiple types of data, such as numerical, textual, geospatial, and so on. The data that flows into Elasticsearch is stored in the form of JSON documents. And related JSON documents can be logically grouped into what is called an index, which allows for fast searching. The plural of index is "indices" and Elasticsearch will use many of these in production workloads. It will also distribute data across multiple servers for better scalability and resilience.

Logstash: It is a tool that takes data/logs from various sources, transforms them, and then stashes them at their destination. We can run these data ingestion pipelines on a real-time basis. A pipeline consists of three stages:

1. Input stage
2. Filter stage
3. Output stage

The input stage can use many different plugins that help it fetch data from multiple types of sources. The filter stage allows it to transform the data before sending it to the destination. For example, some data may be too big, and we only need part of it. Extracting just a small part can greatly reduce our storage needs. Or some data might need to be transformed from text to numbers. Other data should be anonymized, and so on. The output stage stashes the transformed data at the desired destination. We can store the data in various places, such as Elasticsearch, simple files, or we can send it to HTTP endpoints, Kafka, etc.

Kibana:

Kibana is a user interface that helps us view the data stored in Elasticsearch. We can use it to create nice visualizations that make it much easier to spot certain things. For example, imagine we have 500 servers. Showing a bunch of numbers, about how intensively CPU cores are used, and how much RAM is free, will be hard to read through. But displaying some horizontal bars next to each server -- the more RAM that is used, the longer the bar -- now that will make it super easy to spot how heavily each server is used.

Features of ELK

High availability

It ensures that your data is replicated and simultaneously available on multiple servers. Even if one of them goes down, replicas of that data will still be available somewhere else. So things will still work.

Stack security

It provides multiple security features such as role-based access control RBAC, encrypted communication, SSO, IP filtering, etc.

Wide range of integrations

Elasticsearch can be accessed using RESTful APIs so a lot of applications can connect to it and use its search capabilities.

Export data and dashboards

You can export Kibana visualizations wherever you need them. For example, you could export to a web page so your clients can check out the status of various things. Or you can export to a private web page, so that your team can check out the status of various parts of the infrastructure.

Data is easy to export to other formats as well, such as CSV files, which might be needed by other teams, like accountants, to generate their own documents.

Automated alerts

Someone will not be watching those logs at all times. So you can set up alerts that email people, or notify them on Slack channels. This way, someone can step up and fix things as soon as something goes wrong, instead of a problem remaining undetected until somebody takes a look at the proper logs.

Many more cool features are available and if you want to find out about them all, you can check out this web page listing features of the ELK stack.

ELK Stack Playground

Installing and configuring Elasticsearch, Logstash, and Kibana, can be a bit tedious. Your time is precious, so you don't have to go through all of those steps. In this playground, you get access to a complete ELK stack already preconfigured. All you need to access it is your web browser. You'll also find a Kibana Dashboard web user interface easily accessible, so you can experiment with visualizations.

We hope you'll have a great time learning in this environment!