Skip to main content
Start by remembering a simple rule: any context you want attached to a record must be added before Elasticsearch receives it. That means the “stamp” (for example, which environment, team, or node produced the event) belongs in Fluentd — upstream of every output.
A simple architecture diagram titled "The Stamp Belongs in Fluentd" showing node01 sending data to Fluentd, which forwards it to Elasticsearch. A stamp icon and a "USER5 context" tag are shown above, with a user avatar labeled "Bob" on the right.
The timestamp and additional metadata (for example, environment or owning team) should be injected inside Fluentd so every record that leaves has the required context. What does a Fluentd filter plugin look like, and how do you install one?

​
Add a custom filter plugin

Bob creates a plugins folder next to his Fluentd configuration and adds a Ruby file. Fluentd recognizes filter plugins when the filename begins with filter_.
The plugin code registers a filter plugin, declares a configuration parameter, and implements a filter method that runs for every record.
additional_data: 'static_info' is a placeholder. In production use something meaningful: production, team-a, cluster-1, etc. The shape of the plugin remains the same — it simply enriches each record before forwarding. Example record after the filter runs:
But Fluentd will not automatically use the plugin just because the file exists. You must enable it in Fluentd’s configuration.

​
Where the filter sits in fluent.conf

A Fluentd configuration is processed top to bottom. The <source> section ingests events and the <match> section emits them. Filters that mutate records must appear between <source> and <match> to run before the output stage.
A diagram titled "Top to Bottom" showing a fluent.conf file with three stacked blocks labeled <source>, <filter>, and <match>. The Fluentd logo is on the left and a USER5 static_info output badge appears to the right.
Place your filter section like this:
Filters modify records as they pass through Fluentd. Always place filters between the <source> and <match> sections so they run before output.

​
Make the plugin visible to the Fluentd container

The plugin file above lives on the host, but Fluentd runs inside a container. Bob updates the Fluentd DaemonSet pod template to mount the host plugins folder into the container at /fluentd/plugins.
A dark-themed diagram titled "One More Gap" showing a node labeled "node01" with a plugins box and a dashed container holding a small blue mascot-like icon. A blank terminal window is shown to the right, suggesting a gap between plugins and the container.
Example volume mounts (snippet from the DaemonSet pod template):
When using hostPath mounts for plugins, ensure file permissions and SELinux/AppArmor policies allow the Fluentd container to read the plugin files.
After applying the updated DaemonSet, Fluentd pods restart and mount the plugins folder. The pipeline now looks like this: a log line arrives, the filter (the stamp) runs inside Fluentd, and the record exits with additional_data: static_info attached before it reaches Elasticsearch.
Next, configure index lifecycle management (ILM) in Elasticsearch so indices roll over and expire automatically.

​
Apply Index Lifecycle Management (ILM) in Elasticsearch

ILM policies live in Elasticsearch and are created through the ILM API. Bob’s Elasticsearch is reachable at port 30200. He defines a policy with a hot phase that rolls over when the index reaches 5 GB or 30 days, and a delete phase that removes indices older than 90 days:
Summary of ILM phases and thresholds: Bob uploads the policy and verifies Elasticsearch stores it (note Elasticsearch normalizes units such as 5GB to 5gb in responses):
A policy alone does not affect indices until a template or a direct setting references it — it’s just a rulebook.
The image shows a dark slide titled "A Rulebook Nobody Picked Up" with a small panel labeled "my_logs_policy" listing "Rollover – 5 GB" and "Delete – 90 days." A dashed arrow points right from the panel and a KodeKloud logo appears in the bottom-left.

​
Create an index template with a rollover alias

Bob creates an index template my_logs_template that matches Fluentd indices (for example, those beginning with fluentd-), attaches the lifecycle policy my_logs_policy, and configures a rollover alias my-logs. The alias provides a stable write target — Fluentd always writes to my-logs while the underlying indices roll over behind the scenes.
A diagram titled "One Stable Name" with the Fluentd logo on the left and four stacked cylinder storage icons across the bottom. An arrow labeled "my-logs" points to the rightmost, highlighted cylinder to indicate a single stable storage target.
Reading the template back confirms the lifecycle and rollover alias are set:
From now on, new indices created under that template inherit the ILM policy and rollover behavior.

​
The result: searchable, stamped logs and automated retention

Remember the painful nights when a pod had been replaced and logs were gone? With the filter stamping records in Fluentd and ILM in Elasticsearch:
  • Every log line includes where it came from (node, team, environment).
  • Indices roll over automatically and old data is deleted per policy.
  • Kibana queries across nodes become simple: search for user5 and you get results from every node.
  • Elasticsearch can scale and surface alerts during load spikes so incidents are detected and acted on quickly.
A dark-themed dashboard illustration titled "An Answer in Minutes" showing two glowing blue node icons labeled node01 and node02. A circular gauge reads "94% Heap" and there's a user avatar labeled "Bob" on the right.
He can now answer “what happened last night?” in minutes and share a clear incident timeline and root cause with the team.

Watch Video

Practice Lab