Skip to main content
Something broke in Bob’s cluster last night. This morning everyone wants to know what happened. Bob runs the team’s Kubernetes cluster, so the question lands on his desk. At the center of the story is a small pod called app. All day and night it writes a line every time someone does something on the website: a user logs in, another user opens a page, someone’s login fails. Each of those lines is a tiny piece of evidence. Where does Bob go to read last night’s evidence?
The image titled "Last Night's Evidence" shows a smiling app/server labeled "node01" in the center with a user avatar labeled "Bob" to the left. On the right are timestamped log entries: 02:14 USER1 logged in, 02:14 USER2 opened page, and 02:15 USER3 login failed.
His first move is the obvious one: he asks Kubernetes for the pod’s logs:
Those lines are from this morning. The pod that was running at 02:14 isn’t the same pod — Kubernetes replaced it a few hours ago. The old pod’s logs looked like this:
When a pod is replaced, its on-node logs go with it. You can sometimes retrieve the previous container’s logs with kubectl logs -p if the underlying container runtime still exposes them, but that’s not reliable across node failures or long-retained history.
If you need immediate troubleshooting, try kubectl logs -p <pod> to fetch the previous container logs when available. For long-term, cluster-wide investigation you need a centralized logging solution.
A single app is simple, but Bob’s cluster has many nodes and many pods. Each node stores its own log files on disk. The app writes into whatever node it lands on. If the incident affected pods on multiple nodes, Bob would have to SSH into each node, tail log files, copy lines into a notepad, and align timestamps by hand — a hopeless task in the middle of a production outage. Design the solution by working backward from the problem:
  • Each log line must leave the node quickly after it’s written so it survives pod replacements.
  • Logs need a durable, searchable store that keeps them after pods are gone.
  • Operators need a fast, intuitive UI to find relevant lines (for example: search for user1 and see all related events in order).
Who performs each role in the recommended stack?
  • Collector on each node: Fluentd — it tails log files or reads container stdout/stderr, normalizes and enriches records via plugins, then forwards them.
  • Durable store: Elasticsearch — an HTTP-searchable document store where each log line becomes a document (default HTTP port 9200).
  • UI / Window: Kibana — a web UI that queries Elasticsearch and presents search, dashboards, and visualizations (default port 5601).
Put the first letters together and you get the EFK stack.
A dark-themed infographic titled "The Window Is Kibana" showing a log pipeline: a smiling agent sends data to Fluentd (plugins), which forwards to Elasticsearch (port 9200), then to Kibana (port 5601) and finally to a user labeled "Bob," alongside a database and dashboard visualization.
At a glance — what each component does: Why run EFK inside Kubernetes instead of on separate servers? Kubernetes already provides the primitives you need to run these components reliably:
  • Fluentd needs one instance per node — Kubernetes’ DaemonSet is exactly for that: add a node, and a Fluentd pod appears on it.
  • Elasticsearch must retain data across pod restarts — a StatefulSet gives pods stable network IDs and makes pairing with PersistentVolumes straightforward.
  • Kibana is stateless — a simple Deployment is sufficient.
  • A Service gives Fluentd a stable address to reach Elasticsearch even as pods are rescheduled.
Below is a compact mapping of Kubernetes resources to EFK needs.
A dark-themed diagram titled "Kibana and a Service" showing three small blue smiling robot icons sitting on separate platforms, with a labeled "Service" box above the center robot and arrows indicating connections from the left to the service and from the service to the center robot. A KodeKloud logo appears in the lower-left corner.
This lesson will walk through building the stack by hand — writing manifests one file at a time — because authoring the YAML yourself is the best way to learn what every line does. You will:
  1. Deploy Elasticsearch first (create StatefulSet + PersistentVolumes).
  2. Add Fluentd as a DaemonSet to push logs into Elasticsearch.
  3. Deploy Kibana on top so you can search and visualize the logs.
After logs flow, you’ll monitor cluster health, configure alerts, scale Elasticsearch as log volume grows, secure access (RBAC, TLS), and add parsing/enrichment rules in Fluentd.
A dark-themed slide titled "The Build Order" showing a three-layer server stack with icons and labels for Elasticsearch, Fluentd, and Kibana, plus a circular "Health" icon on the right. The KodeKloud logo appears in the bottom-left corner.
Keep one mental picture for the rest of the guide: a line about user1 logging in is born inside the app, Fluentd picks it up on the node, Elasticsearch stores it as a document, and Bob finds it in Kibana minutes or months later. Every log moves that line a little further along its journey. Before adding more pods to the cluster, Bob should inventory what’s already running and confirm node-level access, storage classes, and resource quotas — the configuration choices he makes now will determine how resilient and searchable his logs are later. Links and references

Watch Video