> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Hands On Advanced Configuration

> How to enrich logs in Fluentd with custom filter plugins, mount them in containers, and configure Elasticsearch index lifecycle and rollover policies for automated log retention and searchable records

Start by remembering a simple rule: any context you want attached to a record must be added before Elasticsearch receives it. That means the “stamp” (for example, which environment, team, or node produced the event) belongs in Fluentd — upstream of every output.

![A simple architecture diagram titled "The Stamp Belongs in Fluentd" showing node01 sending data to Fluentd, which forwards it to Elasticsearch. A stamp icon and a "USER5 context" tag are shown above, with a user avatar labeled "Bob" on the right.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Advanced-EFK-Configuration/Hands-On-Advanced-Configuration/stamp-in-fluentd-node01-to-elasticsearch.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=5298141ce4d481440534a3348f7a58bc)

The timestamp and additional metadata (for example, environment or owning team) should be injected inside Fluentd so every record that leaves has the required context.

What does a Fluentd filter plugin look like, and how do you install one?

## Add a custom filter plugin

Bob creates a `plugins` folder next to his Fluentd configuration and adds a Ruby file. Fluentd recognizes filter plugins when the filename begins with `filter_`.

```bash theme={null}
$ mkdir -p /root/fluentd/plugins
$ vi /root/fluentd/plugins/filter_my_custom_filter.rb
```

The plugin code registers a filter plugin, declares a configuration parameter, and implements a `filter` method that runs for every record.

```ruby theme={null}
require 'fluent/plugin/filter'

class Fluent::Plugin::MyCustomFilter < Fluent::Plugin::Filter
  Fluent::Plugin.register_filter('my_custom_filter', self)

  config_param :additional_data, :string, default: 'static_info'

  def filter(tag, time, record)
    record['additional_data'] = @additional_data
    record
  end
end
```

`additional_data: 'static_info'` is a placeholder. In production use something meaningful: `production`, `team-a`, `cluster-1`, etc. The shape of the plugin remains the same — it simply enriches each record before forwarding.

Example record after the filter runs:

```text theme={null}
USER5 failed to log in  additional_data: static_info
environment: production
```

But Fluentd will not automatically use the plugin just because the file exists. You must enable it in Fluentd’s configuration.

## Where the filter sits in fluent.conf

A Fluentd configuration is processed top to bottom. The `<source>` section ingests events and the `<match>` section emits them. Filters that mutate records must appear between `<source>` and `<match>` to run before the output stage.

![A diagram titled "Top to Bottom" showing a fluent.conf file with three stacked blocks labeled \<source>, \<filter>, and \<match>. The Fluentd logo is on the left and a USER5 static\_info output badge appears to the right.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Advanced-EFK-Configuration/Hands-On-Advanced-Configuration/fluentd-top-bottom-source-filter-match.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=8837ea27db7d6c52414449acde468d15)

Place your filter section like this:

```text theme={null}
fluent.conf

<source>
  ...
</source>

<filter **>
  @type my_custom_filter
  additional_data static_info
</filter>

<match **>
  ...
</match>
```

Filters modify records as they pass through Fluentd. Always place filters between the `<source>` and `<match>` sections so they run before output.

## Make the plugin visible to the Fluentd container

The plugin file above lives on the host, but Fluentd runs inside a container. Bob updates the Fluentd DaemonSet pod template to mount the host `plugins` folder into the container at `/fluentd/plugins`.

![A dark-themed diagram titled "One More Gap" showing a node labeled "node01" with a plugins box and a dashed container holding a small blue mascot-like icon. A blank terminal window is shown to the right, suggesting a gap between plugins and the container.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Advanced-EFK-Configuration/Hands-On-Advanced-Configuration/one-more-gap-node01-plugins-container.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=762b3642080a8c622332d1bc2287143e)

Example volume mounts (snippet from the DaemonSet pod template):

```yaml theme={null}
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: fluentd
          volumeMounts:
            - name: configpath
              mountPath: /fluentd/etc
            - name: pluginpath
              mountPath: /fluentd/plugins
      volumes:
        - name: configpath
          hostPath:
            path: /root/fluentd/etc
        - name: pluginpath
          hostPath:
            path: /root/fluentd/plugins
```

When using `hostPath` mounts for plugins, ensure file permissions and SELinux/AppArmor policies allow the Fluentd container to read the plugin files.

After applying the updated DaemonSet, Fluentd pods restart and mount the `plugins` folder. The pipeline now looks like this: a log line arrives, the filter (the stamp) runs inside Fluentd, and the record exits with `additional_data: static_info` attached before it reaches Elasticsearch.

```text theme={null}
USER5 failed to log in  additional_data: static_info
```

Next, configure index lifecycle management (ILM) in Elasticsearch so indices roll over and expire automatically.

## Apply Index Lifecycle Management (ILM) in Elasticsearch

ILM policies live in Elasticsearch and are created through the ILM API. Bob’s Elasticsearch is reachable at port `30200`.

He defines a policy with a `hot` phase that rolls over when the index reaches 5 GB or 30 days, and a `delete` phase that removes indices older than 90 days:

```json theme={null}
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5GB",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
```

Summary of ILM phases and thresholds:

| Phase | Action | Threshold / Setting |
| - | - | - |
| Hot | Rollover | `max_size: "5GB"`, `max_age: "30d"` |
| Delete | Delete indices | `min_age: "90d"` |

Bob uploads the policy and verifies Elasticsearch stores it (note Elasticsearch normalizes units such as `5GB` to `5gb` in responses):

```bash theme={null}
$ curl -X PUT "http://localhost:30200/_ilm/policy/my_logs_policy" \
  -H 'Content-Type: application/json' -d @my_logs_policy.json
{"acknowledged":true}

$ curl -s -X GET "http://localhost:30200/_ilm/policy/my_logs_policy" \
  | jq '.my_logs_policy.policy.phases.hot.actions.rollover.max_size'
"5gb"
```

A policy alone does not affect indices until a template or a direct setting references it — it’s just a rulebook.

![The image shows a dark slide titled "A Rulebook Nobody Picked Up" with a small panel labeled "my\_logs\_policy" listing "Rollover – 5 GB" and "Delete – 90 days." A dashed arrow points right from the panel and a KodeKloud logo appears in the bottom-left.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Advanced-EFK-Configuration/Hands-On-Advanced-Configuration/rulebook-my-logs-policy-5gb-90days.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=39d612a1aa70e958d6289efae8dcf074)

## Create an index template with a rollover alias

Bob creates an index template `my_logs_template` that matches Fluentd indices (for example, those beginning with `fluentd-`), attaches the lifecycle policy `my_logs_policy`, and configures a rollover alias `my-logs`. The alias provides a stable write target — Fluentd always writes to `my-logs` while the underlying indices roll over behind the scenes.

![A diagram titled "One Stable Name" with the Fluentd logo on the left and four stacked cylinder storage icons across the bottom. An arrow labeled "my-logs" points to the rightmost, highlighted cylinder to indicate a single stable storage target.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Advanced-EFK-Configuration/Hands-On-Advanced-Configuration/fluentd-my-logs-stable-storage.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=d3c3d05127230190296c2f84534244ec)

Reading the template back confirms the lifecycle and rollover alias are set:

```bash theme={null}
$ curl -s -X GET "http://localhost:30200/_template/my_logs_template" \
  | jq '.my_logs_template.settings.index'
{
  "lifecycle": {
    "name": "my_logs_policy",
    "rollover_alias": "my-logs"
  }
}
```

From now on, new indices created under that template inherit the ILM policy and rollover behavior.

## The result: searchable, stamped logs and automated retention

Remember the painful nights when a pod had been replaced and logs were gone? With the filter stamping records in Fluentd and ILM in Elasticsearch:

* Every log line includes where it came from (node, team, environment).
* Indices roll over automatically and old data is deleted per policy.
* Kibana queries across nodes become simple: search for `user5` and you get results from every node.
* Elasticsearch can scale and surface alerts during load spikes so incidents are detected and acted on quickly.

![A dark-themed dashboard illustration titled "An Answer in Minutes" showing two glowing blue node icons labeled node01 and node02. A circular gauge reads "94% Heap" and there's a user avatar labeled "Bob" on the right.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Advanced-EFK-Configuration/Hands-On-Advanced-Configuration/dark-dashboard-node01-node02-heap94-bob.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=9392842e5d47320bebfecb9ef430f98b)

He can now answer “what happened last night?” in minutes and share a clear incident timeline and root cause with the team.

## Links and references

* Fluentd Filter Plugin docs: [https://docs.fluentd.org/plugin-development](https://docs.fluentd.org/plugin-development)
* Elasticsearch ILM: [https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)
* Kubernetes DaemonSet: [https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)

> [card] [Watch Video](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/0e6af297-267f-4e2b-8929-b34e42dae3eb/lesson/c0917462-f21c-4ae6-b937-ffc6e410d246)

> [card] [Practice Lab](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/0e6af297-267f-4e2b-8929-b34e42dae3eb/lesson/e130b8b7-6009-44b0-a79e-cfc6f7d9c42b)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.