Hands On Fluentd Deployment

Time to deploy the collector for real. This guide follows Bob’s steps to install Fluentd as a DaemonSet in the elastic-stack namespace so every node ships container logs to Elasticsearch.

Context: where we are

Bob starts in the elastic-stack namespace, where two pods already run: one app that writes user events and elasticsearch-0. Everything we create goes into that same namespace. Create a working folder on the host to hold the Fluentd configuration that will be mounted into each DaemonSet pod:

$ mkdir -p /root/fluentd

Why a host folder? The DaemonSet will mount this host path into every Fluentd pod so each instance reads the same fluent.conf you place there.

Who is Fluentd to Kubernetes?

Fluentd needs to ask the Kubernetes API server about pods and namespaces (so it can discover containers to tail). That means you must give Fluentd an identity and the minimal permissions it needs.

RBAC objects to create

Bob creates three small RBAC objects:

A ServiceAccount named fluentd in elastic-stack (the pod identity).
A ClusterRole named fluentd granting read-only access to pods and namespaces.
A ClusterRoleBinding that binds the ClusterRole to the ServiceAccount.

These three objects together let any pod using the fluentd ServiceAccount query the API server to list and watch pods and namespaces — and nothing else.

Resource	Purpose	Example / Command
ServiceAccount	Pod identity used to authenticate to API server	`fluentd-sa.yaml` → `kubectl apply -f fluentd-sa.yaml`
ClusterRole	Read-only permissions (`get`, `list`, `watch`) on `pods` and `namespaces`	`fluentd-clusterrole.yaml` → `kubectl apply -f fluentd-clusterrole.yaml`
ClusterRoleBinding	Binds the ClusterRole to the ServiceAccount	`fluentd-clusterrolebinding.yaml` → `kubectl apply -f fluentd-clusterrolebinding.yaml`

Example commands Bob ran:

$ kubectl apply -f fluentd-sa.yaml
serviceaccount/fluentd created

$ kubectl get serviceaccount fluentd -n elastic-stack
NAME       SECRETS   AGE
fluentd    0         9s

$ kubectl apply -f fluentd-clusterrole.yaml
clusterrole.rbac.authorization.k8s.io/fluentd created

$ kubectl get clusterrole fluentd
NAME      CREATED AT
fluentd   2026-10-13T08:41:07Z

$ kubectl apply -f fluentd-clusterrolebinding.yaml
clusterrolebinding.rbac.authorization.k8s.io/fluentd created

$ kubectl get clusterrolebinding fluentd
NAME      ROLE                   AGE
fluentd   ClusterRole/fluentd    4s

Fluentd configuration: how a log line flows

Create the config folder on the host, and create fluent.conf inside it:

$ mkdir -p /root/fluentd/etc
$ touch /root/fluentd/etc/fluent.conf

The DaemonSet will mount /root/fluentd/etc from the host into /fluentd/etc in each pod, so whatever you put there is what Fluentd reads on startup. Below is a walkthrough of the main pieces in fluent.conf, in the order a log line meets them:

source: tails files under /var/log/containers/*.log (container runtime symlinks). A small pos_file ensures Fluentd resumes where it left off after a restart. Fluentd excludes its own container logs to avoid self-collection and sets read_from_head so it reads existing files from the beginning.
parse: a nested <parse> block uses @type json to parse each line produced by the container runtime into fields such as time, stream, and log.
match: captures all records and forwards them to Elasticsearch. Documents are written in Logstash format to an index prefixed fluentd-YYYY.MM.DD. A buffer flushes every 5s and retry_forever true retains records until Elasticsearch accepts them.

Example fluent.conf (tailing Kubernetes container logs and shipping to Elasticsearch):

<source>
  @type tail
  path "/var/log/containers/*.log"
  pos_file "/var/log/fluentd-containers.log.pos"
  tag "kubernetes.*"
  exclude_path /var/log/containers/fluent*
  read_from_head true

  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

<match **>
  @type elasticsearch
  host "elasticsearch.elastic-stack.svc.cluster.local"
  port 9200
  logstash_format true
  logstash_prefix "fluentd"
  include_tag_key true
  type_name "_doc"

  <buffer>
    flush_interval 5s
    retry_forever true
  </buffer>
</match>

A sample container log line this config parses might look like:

{
  "time": "2026-10-13T08:12:04.512Z",
  "stream": "stdout",
  "log": "USER1 logged in"
}

DaemonSet manifest highlights

Bob deploys Fluentd as a DaemonSet so one collector runs on each node and tails logs from the node filesystem. Key manifest decisions:

Image: a Fluentd build for Kubernetes + Elasticsearch 7: fluent/fluentd-kubernetes-daemonset:v1.14.1-debian-elasticsearch7-1.0
Environment variables configure Elasticsearch host, port, scheme, and Logstash prefix.
A toleration allows scheduling on control-plane nodes (where workloads may also run).
Important host mounts:
- /var/log (node logs)
- /var/lib/docker/containers (container runtime files, read-only)
- host path /root/fluentd/etc → /fluentd/etc (Fluentd config)
Resource limit: memory 512Mi to avoid a collector consuming an entire node.

Relevant DaemonSet fragments:

tolerations:
  - key: node-role.kubernetes.io/master
    effect: NoSchedule

env:
  - name: FLUENT_ELASTICSEARCH_HOST
    value: "elasticsearch.elastic-stack.svc.cluster.local"
  - name: FLUENT_ELASTICSEARCH_PORT
    value: "9200"
  - name: FLUENT_ELASTICSEARCH_SCHEME
    value: "http"
  - name: FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX
    value: "fluentd"

resources:
  limits:
    memory: 512Mi

volumeMounts:
  - name: varlog
    mountPath: /var/log
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true
  - name: configpath
    mountPath: /fluentd/etc

volumes:
  - name: configpath
    hostPath:
      path: /root/fluentd/etc

Apply the DaemonSet:

$ kubectl apply -f fluentd.yaml
daemonset.apps/fluentd created

$ kubectl get ds fluentd -n elastic-stack -o jsonpath='{.spec.template.spec.containers[0].image}'
fluent/fluentd-kubernetes-daemonset:v1.14.1-debian-elasticsearch7-1.0

The jsonpath check verifies the exact image on the DaemonSet template so typos are easy to catch. Kubernetes then schedules a Fluentd pod onto each node. Each pod runs with the fluentd ServiceAccount and uses the mounted host config.

Verifying logs reach Elasticsearch

After Fluentd pods are running, confirm documents arrive in Elasticsearch and that they contain your app logs (log field). From a pod in the same namespace you can query the cluster IP service: List indices to confirm fluentd-* exists:

$ kubectl run -n elastic-stack --rm -it curl-debug --image=curlimages/curl --restart=Never -- \
  curl -s http://elasticsearch.elastic-stack.svc.cluster.local:9200/_cat/indices?v

Inspect the most recent document to verify structure and content (look for log containing app output such as USER1 logged in):

$ kubectl run -n elastic-stack --rm -it curl-debug --image=curlimages/curl --restart=Never -- \
  curl -s 'http://elasticsearch.elastic-stack.svc.cluster.local:9200/fluentd-*/_search?size=1&sort=@timestamp:desc' | jq .

If you see fluentd-* indices and documents with the log field containing your application messages, Fluentd is correctly tailing container logs and shipping them to Elasticsearch.

If Elasticsearch is not reachable immediately after deploying Fluentd, allow a short time for Fluentd to establish connections and for buffers to flush. The retry_forever true buffer setting ensures events are retried until delivery succeeds.

Quick checklist

Host folder created: /root/fluentd/etc/fluent.conf
ServiceAccount fluentd created and used by the DaemonSet
ClusterRole grants get,list,watch on pods and namespaces
ClusterRoleBinding binds the role to the fluentd ServiceAccount
DaemonSet mounts node log paths and the host config path
Fluentd pods are running on each node
Check Elasticsearch for fluentd-* indices and confirm the log field contains app messages

References

Kubernetes RBAC documentation: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
Fluentd Kubernetes DaemonSet images: https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset
Elasticsearch HTTP API: https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html

Introduction to EFK on Kubernetes

Deploying Elasticsearch

Collecting Logs With Fluentd

Visualizing Logs With Kibana

Monitoring and Alerting

Scaling the EFK Stack

Securing the EFK Stack

Advanced EFK Configuration

Hands On Fluentd Deployment

Context: where we are

Who is Fluentd to Kubernetes?

RBAC objects to create

Fluentd configuration: how a log line flows

DaemonSet manifest highlights

Verifying logs reach Elasticsearch

Quick checklist

References

Watch Video

​Context: where we are

​Who is Fluentd to Kubernetes?

​RBAC objects to create

​Fluentd configuration: how a log line flows

​DaemonSet manifest highlights

​Verifying logs reach Elasticsearch

​Quick checklist

​References

Watch Video

Context: where we are

Who is Fluentd to Kubernetes?

RBAC objects to create

Fluentd configuration: how a log line flows

DaemonSet manifest highlights

Verifying logs reach Elasticsearch

Quick checklist

References