> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Fluentd Problem and Solution
> Explains deploying Fluentd as a Kubernetes DaemonSet to tail host container logs, parse and enrich them, and forward to Elasticsearch, including mounts and RBAC considerations
Elasticsearch is up and listening on port 9200. It has its own storage, and for a brief moment Bob is happy.
Then Bob asks Elasticsearch for every document it has — and the response is empty. Zero hits.
Meanwhile, the application is busy on its node, continuously writing lines about user activity: "something user 1 did", "something user 3 did". The logs exist, but they're on the nodes where containers run — not in Elasticsearch. Elasticsearch is a store; it will only contain data that someone forwards to it.
So where are those log lines right now?
When a container prints a log line, the container runtime persists it on the host node. Typically you’ll find container logs under `/var/log/containers` (one file per container is common, though exact layout varies by runtime and host). In short: logs accumulate on the node where the container happens to run. Elasticsearch will not “go looking” for them — something else must collect and send them.
You might imagine Bob writing a cron script that copies log files from every node to a central place. That can work temporarily, but it breaks when pods are rescheduled to other nodes or when new nodes are added and the script wasn't deployed there.
Because log files are born on every node, the collector that reads them must run on every node as well. The standard solution in Kubernetes is Fluentd.
What Fluentd does
* Fluentd tails log files on the node, parses and structures each line, optionally filters or enriches events, and forwards them to outputs such as Elasticsearch.
* Picture Fluentd running on the node, watching the app’s log file. New lines appear; Fluentd reads them and immediately ships them to Elasticsearch on port 9200.
Why a Deployment is not enough
A Kubernetes Deployment requests a number of replicas (for example, two) and the scheduler places them wherever it chooses — potentially co-locating both pods on the same node. That does not guarantee exactly one copy per node.
Use a DaemonSet instead
Kubernetes provides the DaemonSet object for exactly this requirement: one pod on every node. Describe the Fluentd pod once in a DaemonSet and Kubernetes ensures a Fluentd pod runs on each node (including nodes that join later). Learn more: [DaemonSet documentation](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/).
How Fluentd reads host logs
Because Fluentd runs inside a container, it accesses host log files by mounting host directories into the pod. Typical mounts include `/var/log` and the container runtime’s directory (paths vary with runtime). Mounts are usually read-only for safety.
Example mounts:
```yaml theme={null}
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
```
Common runtime log locations
| Container runtime | Typical log path(s) on host |
| - | - |
| Docker | `/var/lib/docker/containers` |
| containerd / CRI-O | `/var/log/pods`, `/var/log/containers` |
| Kubernetes distributions | Paths may vary; check your distro docs |
Fluentd can read these host files but should not modify them.
Fluentd configuration basics
Fluentd’s behavior is defined by directive blocks. The main building blocks are:
* source — where records come from (e.g., tailing files)
* parse — how to convert raw text into structured fields
* filter — optional event transformation, enrichment, or dropping
* match — route records to outputs (e.g., Elasticsearch)
Example: tail every container log in `/var/log/containers` and parse each line
```xml theme={null}
@type tail
@id in_tail_container_logs
path "/var/log/containers/*.log"
pos_file "/var/log/fluentd-containers.log.pos"
tag "kubernetes.*"
exclude_path /var/log/containers/fluent*
read_from_head true
@type regexp
expression /^(?
```
A raw log line might look like:
```text theme={null}
2024-04-15T05:12:09Z stdout USER1 logged in
```
The parser in the example extracts named fields: `time`, `stream`, and `log`. With structured records, Fluentd can route, enrich, or filter events more effectively. If logs include fractional seconds or a different timestamp format, adjust `time_format` accordingly.
Filtering and enrichment
After parsing you can add filter blocks (for example, to add Kubernetes metadata by querying the API server, or to drop noisy messages). Filters are commonly used in production pipelines to reduce storage usage and improve searchability.
Sending logs to Elasticsearch
A match block sends records to an output plugin. Here’s an example using the Elasticsearch output plugin and pointing to an in-cluster service:
```xml theme={null}
@type elasticsearch
@id out_es
@log_level info
include_tag_key true
host "elasticsearch.elastic-stack.svc.cluster.local"
port 9200
```
RBAC: granting Fluentd access to cluster metadata
Fluentd commonly enriches logs with Kubernetes metadata (pod name, namespace, labels). To read the Kubernetes API you must give Fluentd an identity and permissions:
* Create a ServiceAccount for Fluentd
* Create a ClusterRole (or Role) with `get`, `list`, `watch` permissions for pods and namespaces
* Bind the role to the ServiceAccount with a ClusterRoleBinding or RoleBinding
This RBAC configuration allows Fluentd to query the API server securely and enrich events with pod/namespace metadata.
Outcome for operators
Before Fluentd: to find last night’s error you had to guess the node, SSH into it, and tail a file that might have rotated away. With Fluentd and a central store (Elasticsearch), each line leaves the node seconds after it’s written, searchable from one place — Bob no longer needs to visit nodes to debug issues.
Summary / Next steps
* Use a DaemonSet to ensure Fluentd runs on every node.
* Mount the host log directories into the Fluentd pod so it can tail container logs.
* Configure source → parse → (optional) filter → match to route logs to Elasticsearch.
* Grant Fluentd a ServiceAccount with RBAC permissions to enrich logs using the Kubernetes API.
Keep in mind: log file paths and formats can vary by container runtime (Docker, containerd, CRI-O) and Kubernetes distributions. Adjust the `path` and parser accordingly when deploying Fluentd in your cluster. For more details, see the [Fluentd docs](https://docs.fluentd.org/) and the [Kubernetes logging concepts](https://kubernetes.io/docs/concepts/cluster-administration/logging/).
> [card] [Watch Video](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/b47732d3-af85-4c4a-8316-6c6ae141d927/lesson/d1100bc7-2616-4541-b64d-5022d64b85c4)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.