Skip to main content
Time to deploy the Fluentd collector into the cluster. Bob starts by verifying his working namespace is elastic-stack. Two pods already run there: an app pod producing user events, and elasticsearch-0 (empty for now). All resources we create will live in the elastic-stack namespace. Create a local workspace for Fluentd configuration:
Fluentd needs an identity to query the Kubernetes API (to discover pods and namespaces). That identity is a ServiceAccount. Bob creates a fluentd ServiceAccount in the elastic-stack namespace with the label app: fluentd.
The image is a slide titled "Who Is Fluentd" showing a cartoon teal Fluentd mascot with a question mark, connected by a dashed arrow to an "API server" icon. Below the server are small labels for "Pods" and "Namespaces."
Apply the ServiceAccount manifest and verify it exists:
A ServiceAccount alone has no permissions. Next Bob creates cluster-wide read-only permissions so Fluentd can list and watch pods and namespaces across the cluster. Apply the ClusterRole:
Then bind that ClusterRole to the ServiceAccount with a ClusterRoleBinding:
These three objects form the RBAC setup for Fluentd:
ServiceAccounts identify pods to the API server. ClusterRoles grant permissions cluster-wide; ClusterRoleBindings attach those roles to ServiceAccounts (or users/groups). Use the minimal set of verbs and resources necessary to reduce blast radius.
With identity and permissions configured, Bob prepares Fluentd’s configuration files. He creates an etc directory under /root/fluentd because the DaemonSet will mount this folder into each Fluentd pod.
Walkthrough: how a log line flows through Fluentd (configuration excerpts follow in the order a log is processed).
  1. Source — tail container log files under /var/log/containers/*.log.
    • A pos_file keeps read offsets so Fluentd resumes correctly after restarts.
    • Exclude Fluentd’s own logs to avoid self-collection.
    • read_from_head true ensures Fluentd reads existing files the first time it runs.
  1. Parse — the <parse> block above converts each raw JSON log line into fields such as time, stream, and log. A typical parsed log record looks like:
  1. Match — send parsed records to Elasticsearch.
    • logstash_format true and logstash_prefix "fluentd" write documents to indices like fluentd-YYYY.MM.DD.
    • The <buffer> section accumulates records and flushes them every 5 seconds. If Elasticsearch is unavailable, retry_forever true ensures records remain buffered until delivery succeeds.
DaemonSet: run a Fluentd pod on every node so logs are collected locally. Important DaemonSet highlights:
  • Namespace: elastic-stack
  • Uses the fluentd ServiceAccount
  • Fluentd image built for Elasticsearch 7
  • Environment variables control Elasticsearch connection and index prefix
  • Toleration to allow running on control-plane nodes if necessary
  • HostPath mounts: /var/log, /var/lib/docker/containers (read-only), and the local /root/fluentd/etc mounted into the pod at /fluentd/etc
  • Memory limit: 512Mi
Example snippet from the DaemonSet manifest:
Using hostPath mounts exposes node filesystem contents to the pod. Ensure you trust the container image and limit permissions (e.g., use read-only mounts where possible). Apply least-privilege principles for RBAC and pod security.
Apply the DaemonSet and confirm the image is exactly what you expect (typos in the image string will be obvious):
Now the Fluentd pods should be running on each node. The next step is to confirm logs are arriving in Elasticsearch.
The image shows two cute blue mascot icons sitting on server blocks labeled "controlplane" and "node01" under the title "Is Anything Arriving?". An arrow points from the nodes to an Elasticsearch logo on the right.
If documents appear in Elasticsearch, Bob will inspect the documents’ contents and timestamps to ensure they contain the application’s log lines (and not unrelated system logs). Useful next checks:
  • Query the index pattern fluentd-* in Kibana or via curl to confirm documents exist.
  • Inspect a sample document for expected log, stream, and kubernetes.* metadata.
  • Verify timestamps align with the application events.
References and further reading:

Watch Video