Skip to main content
Elasticsearch is up and listening on port 9200. It has its own storage, and for a brief moment Bob is happy.
A dark teal infographic titled "Zero Hits" showing a smiling blue robot next to a server/database stack with a "9200" label, and a circular user avatar labeled "Bob" on the right. The KodeKloud logo appears in the bottom-left.
Then Bob asks Elasticsearch for every document it has — and the response is empty. Zero hits. Meanwhile, the application is busy on its node, continuously writing lines about user activity: “something user 1 did”, “something user 3 did”. The logs exist, but they’re on the nodes where containers run — not in Elasticsearch. Elasticsearch is a store; it will only contain data that someone forwards to it. So where are those log lines right now?
A diagram titled "The Lines Are Elsewhere" showing four friendly blue robot-like nodes on platforms, each with a document icon and dashed arrows pointing to a rightmost node that’s attached to a database. It appears to illustrate distributed agents sending data toward a central collector.
When a container prints a log line, the container runtime persists it on the host node. Typically you’ll find container logs under /var/log/containers (one file per container is common, though exact layout varies by runtime and host). In short: logs accumulate on the node where the container happens to run. Elasticsearch will not “go looking” for them — something else must collect and send them. You might imagine Bob writing a cron script that copies log files from every node to a central place. That can work temporarily, but it breaks when pods are rescheduled to other nodes or when new nodes are added and the script wasn’t deployed there.
A dark-themed infographic titled "A Copy Script Breaks" with stylized platform icons, a smiling blue bot in the center, and a "copy script" label on the left. On the right is a circular user avatar labeled "Bob" and a cylindrical database icon, with "Wednesday" shown below.
Because log files are born on every node, the collector that reads them must run on every node as well. The standard solution in Kubernetes is Fluentd. What Fluentd does
  • Fluentd tails log files on the node, parses and structures each line, optionally filters or enriches events, and forwards them to outputs such as Elasticsearch.
  • Picture Fluentd running on the node, watching the app’s log file. New lines appear; Fluentd reads them and immediately ships them to Elasticsearch on port 9200.
A dark teal diagram titled "What Fluentd Does" showing two smiling blue robot icons on a platform with a document icon and a dashed arrow labeled "USER3" between them, and an Elasticsearch cylinder pictured to the right.
Why a Deployment is not enough A Kubernetes Deployment requests a number of replicas (for example, two) and the scheduler places them wherever it chooses — potentially co-locating both pods on the same node. That does not guarantee exactly one copy per node.
A diagram titled "A Deployment Won't Do" showing three server nodes labeled node01, node02, and node03 with two blue container/whale icons above and a dashed placeholder for a missing fourth node. The word "Deployment" is shown below.
Use a DaemonSet instead Kubernetes provides the DaemonSet object for exactly this requirement: one pod on every node. Describe the Fluentd pod once in a DaemonSet and Kubernetes ensures a Fluentd pod runs on each node (including nodes that join later). Learn more: DaemonSet documentation. How Fluentd reads host logs Because Fluentd runs inside a container, it accesses host log files by mounting host directories into the pod. Typical mounts include /var/log and the container runtime’s directory (paths vary with runtime). Mounts are usually read-only for safety. Example mounts:
Common runtime log locations Fluentd can read these host files but should not modify them. Fluentd configuration basics Fluentd’s behavior is defined by directive blocks. The main building blocks are:
  • source — where records come from (e.g., tailing files)
  • parse — how to convert raw text into structured fields
  • filter — optional event transformation, enrichment, or dropping
  • match — route records to outputs (e.g., Elasticsearch)
Example: tail every container log in /var/log/containers and parse each line
A raw log line might look like:
The parser in the example extracts named fields: time, stream, and log. With structured records, Fluentd can route, enrich, or filter events more effectively. If logs include fractional seconds or a different timestamp format, adjust time_format accordingly. Filtering and enrichment After parsing you can add filter blocks (for example, to add Kubernetes metadata by querying the API server, or to drop noisy messages). Filters are commonly used in production pipelines to reduce storage usage and improve searchability. Sending logs to Elasticsearch A match block sends records to an output plugin. Here’s an example using the Elasticsearch output plugin and pointing to an in-cluster service:
RBAC: granting Fluentd access to cluster metadata Fluentd commonly enriches logs with Kubernetes metadata (pod name, namespace, labels). To read the Kubernetes API you must give Fluentd an identity and permissions:
  • Create a ServiceAccount for Fluentd
  • Create a ClusterRole (or Role) with get, list, watch permissions for pods and namespaces
  • Bind the role to the ServiceAccount with a ClusterRoleBinding or RoleBinding
This RBAC configuration allows Fluentd to query the API server securely and enrich events with pod/namespace metadata.
A dark infographic titled "Permission to Read" showing an RBAC flow: a cute service-account-like icon on the left with a dashed arrow to cluster role/API icons and then to pod/namespace icons on the right. Labels visible include "sa", "c.role", "api", "pod", and "ns".
Outcome for operators Before Fluentd: to find last night’s error you had to guess the node, SSH into it, and tail a file that might have rotated away. With Fluentd and a central store (Elasticsearch), each line leaves the node seconds after it’s written, searchable from one place — Bob no longer needs to visit nodes to debug issues.
A dark-themed infographic titled "Bob Stops Visiting Nodes" showing a Bob avatar on the left and three server nodes with smiling robot icons across the bottom. A database labeled "USER1" is on the right and the word "Now" is centered at the bottom.
Summary / Next steps
  • Use a DaemonSet to ensure Fluentd runs on every node.
  • Mount the host log directories into the Fluentd pod so it can tail container logs.
  • Configure source → parse → (optional) filter → match to route logs to Elasticsearch.
  • Grant Fluentd a ServiceAccount with RBAC permissions to enrich logs using the Kubernetes API.
Keep in mind: log file paths and formats can vary by container runtime (Docker, containerd, CRI-O) and Kubernetes distributions. Adjust the path and parser accordingly when deploying Fluentd in your cluster. For more details, see the Fluentd docs and the Kubernetes logging concepts.

Watch Video