Advanced Configuration Problem and Solution

time 02:14:07
stream stdout
log USER5 failed to log in, account locked

​

Log Enrichment: Filters and Plugins

<filter **>
  @type my_custom_filter
</filter>

# my_custom_filter.rb
require 'fluent/plugin/filter'

module Fluent::Plugin
  class MyCustomFilter < Fluent::Plugin::Filter
    Fluent::Plugin.register_filter('my_custom_filter', self)

    config_param :team, :string, default: 'unknown'
    config_param :environment, :string, default: 'production'

    def filter(tag, time, record)
      record['team'] ||= @team
      record['environment'] ||= @environment
      record
    end
  end
end

<filter **>
  @type my_custom_filter
  team ops
  environment production
</filter>

​

Index Lifecycle: Rollover and Retention

• Rollover when an index reaches 5 GB or 30 days (whichever comes first) — the “hot” phase.
• Delete indices once they are 90 days old — the “delete” phase.

• Hot phase: the index is actively written to.
• Rollover: switch from the current concrete index (e.g., fluentd-000001) to a new one when size/age limits are reached.
• Delete phase: remove indices after retention expires.
• Use a write alias (for example fluentd-write) that points to the current write index — ILM manages the concrete indices beneath that alias.

1. If you need context that isn’t in the original log line, add it in Fluentd while the event flows through the pipeline — you cannot reconstruct it reliably later.
2. If data has a known shelf life, encode that shelf life in an ILM policy rather than relying on manual deletions.

​

Implementation: Enrichment + ILM (Step-by-step)

1. Create the Fluentd filter plugin and enable it

• Place the Ruby plugin file (example above) into Fluentd’s plugin directory, for example /etc/fluent/plugin/my_custom_filter.rb.
• Add a filter block to Fluentd config to enable the plugin and pass parameters:

<filter **>
  @type my_custom_filter
  team ops
  environment production
</filter>

2. Create an ILM policy in Elasticsearch

• Define a lifecycle policy that rollovers at 5 GB or 30 days and deletes at 90 days:

PUT _ilm/policy/fluentd-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

3. Configure an index template and write alias for rollover

• ILM rollovers work best when writing to an alias like fluentd-write. Attach the ILM policy via an index template and define the alias:

PUT _template/fluentd-template
{
  "index_patterns": ["fluentd-*"],
  "settings": {
    "index.lifecycle.name": "fluentd-policy",
    "index.lifecycle.rollover_alias": "fluentd-write",
    "number_of_shards": 1
  },
  "aliases": {
    "fluentd-write": {}
  }
}

• Create the initial concrete index and mark the alias as the write index:

PUT /fluentd-000001
{
  "aliases": {
    "fluentd-write": {
      "is_write_index": true
    }
  }
}

4. Point Fluentd at the write alias

• Configure Fluentd’s Elasticsearch output to write to the alias name so ILM can rollover the underlying concrete indices:

<match **>
  @type elasticsearch
  host elasticsearch.default.svc.cluster.local
  port 9200
  index_name fluentd-write
  logstash_format false
  type_name _doc
</match>

• Fluentd enriches each record as it passes through the filter, so you can query by team or environment in Kibana and find which copy produced the locked-account message.
• Elasticsearch ILM rollovers indices and deletes them after the retention period, preventing disk-full surprises and removing the need for manual index cleanup.

​

Quick reference

​

Summary

• Use Fluentd filters or custom plugin filters to enrich logs as they pass through the pipeline — add team, environment, pod names, etc. at ingestion time.
• Use Elasticsearch ILM (rollover + delete) to automate lifecycle management and avoid manual deletions and disk exhaustion.
• Combine both: enrichment ensures logs have the context needed to answer operational questions; ILM ensures logs don’t consume unbounded storage.

​

Links and References

• Fluentd documentation: https://docs.fluentd.org/
• Elasticsearch Index Lifecycle Management: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html
• Elasticsearch templates and aliases: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html