K8s with EFK Playground

What is EFK?

EFK stands for Elasticsearch-Fluentd-Kibana. This stack is commonly used for the aggregation of logs and their analysis. What is the role of each of the tools in this stack? 

Elasticsearch is an open-source and distributed search engine that can store data from various sources. It stores the data as JSON documents. A collection of such documents is called an index in Elasticsearch. Once the data is stored, users can search over it for better analysis.

To search over the data, we first need to ingest and store it into Elasticsearch. This is done by Fluentd. Fluentd is responsible for fetching data from different sources. It reads the logs of multiple apps and then stores them in Elasticsearch. It can also extract some specific information from the data.

Now that we are sending all the logs to Elasticsearch using Fluentd, we require a user interface to see the data. Kibana provides a user-friendly UI to view the data. Users can run complex queries using Kibana and perform visual analysis of their logs. For example, bar charts, pie charts, and other types of visual representations make it easy to analyze huge amounts of data collected from multiple sources.

Why use EFK?

It is necessary to monitor the logs of the running apps. These logs tell you if you are getting any errors in those apps and they can often help when debugging those problems as well. Normally the users can access the logs of such apps using the ‘kubectl logs’ command. But the apps can be running on several pods. It becomes difficult to manually check the logs of each and every pod. Also, it may happen that the issue is not at the application end. Some other components might be failing to cause the apps to behave weirdly. In that scenario, you will also need to analyze the logs of those components. Thus, a centralized logging system is necessary to monitor the logs from multiple sources. This ensures that all the logs are getting stored at a central location and users can view them collectively. It provides better troubleshooting and faster resolution.

K8s with EFK playground

The playground offers a pre-configured EFK stack running on top of Kubernetes. One sample app has been deployed on Kubernetes and its logs are exported to Elasticsearch. Users can connect to Kibana UI for viewing these logs. Experiment with the Kibana application and learn how to create the visualizations you need.