Skip to main content
All right — let’s design a resilient centralized logging pipeline for a system running more than one hundred microservices. The goal: reliable collection, durable buffering, scalable processing, and cost-efficient long-term retention so a “noisy” service cannot take the logging stack down. Example of a noisy service producing many errors:
High-level goals
  • Make ingestion resilient to spikes.
  • Keep recent logs fast to search for on-call and alerts.
  • Store all logs durably and cheaply for compliance and deep analysis.
  • Maintain low per-node resource usage for log collectors.
Architecture overview — high-level flow
  • Hundreds of microservices run on Kubernetes.
  • Applications emit structured JSON logs to stdout.
  • The container runtime / kubelet writes container stdout/stderr to node log files.
  • Fluent Bit runs as a DaemonSet on each node, tails container logs, and forwards them in batches.
  • Kafka is used as a durable buffer to decouple producers and consumers.
  • A stream processor (Logstash, Vector, Kafka Connect, or Kafka Streams) consumes from Kafka, parses and enriches logs, tags errors, and writes to:
    • Elasticsearch (hot tier) for fast search and alerting.
    • S3 (cold tier) for long-term, cost-effective retention.
  • Kibana (or equivalent) provides interactive search and dashboards; an alerting engine monitors error patterns and fires alerts.
Why structured logs?
  • JSON logs make it straightforward to index and query fields such as level, service, trace_id, request_id, and timestamp.
  • Structured logs enable precise queries, reduce costly regex parsing, and improve enrichment and correlation across traces and metrics.
Example: application emitting structured JSON to stdout
Collection on Kubernetes
  • Deploy Fluent Bit as a DaemonSet so one lightweight collector runs per node. Fluent Bit has a small memory footprint and is designed for tailing files and forwarding logs efficiently.
  • Fluent Bit tails CRI log files (commonly under /var/log/containers or CRI-managed paths), enriches with basic container metadata, batches messages, and forwards to Kafka.
Important design decision: use Kafka as a buffer
  • Avoid sending logs directly from Fluent Bit to Elasticsearch. Forwarding straight to Elasticsearch couples ingestion spikes to the indexing tier and can overload it during noisy incidents.
Kafka decouples producers and consumers — allowing Elasticsearch and processors to catch up without being overwhelmed.
Do not forward high-volume logs directly to Elasticsearch without an intermediary buffer. Indexing systems are not designed to absorb sudden, sustained ingestion spikes and can become unavailable.
Stream processing, enrichment, and tagging
  • A stream processor reads from Kafka and performs:
    • Parsing (if logs are plain text) or validation of JSON.
    • Enrichment with Kubernetes pod metadata (via the Kubernetes API or metadata plugins).
    • Trace correlation: attach trace_id or join with trace/span data when available.
    • Classification: mark error/severity, extract HTTP status, request path, user id, etc.
    • Routing: send enriched logs to both hot and cold sinks.
  • The stream processor should be horizontally scalable and able to checkpoint offsets so it can resume safely after restarts.
Hot vs cold storage — purpose and retention
  • Hot tier (Elasticsearch / OpenSearch)
    • Purpose: low-latency search, dashboards, and alerting for recent incidents.
    • Typical retention: ~7 days (customizable based on SLOs and cost).
  • Cold tier (S3 or object storage)
    • Purpose: cost-effective long-term retention for compliance, audits, and historical analysis.
    • Typical retention: ~1 year or more depending on compliance requirements.
Recommended component mapping Retention and lifecycle (example policy) Putting the pieces together — why this pipeline survives a “bad day”
  • Fluent Bit keeps collection efficient and low-overhead on each node.
  • Kafka buffers ingestion spikes and provides durable storage so downstream consumers can process at their own pace.
  • Stream processors enrich and route logs to both a low-latency hot tier (for alerting) and a durable cold tier (for retention).
  • Separation of concerns and buffering ensure that a single noisy service cannot overwhelm Elasticsearch or the UI.
Operational recommendations
  • Monitor key metrics for each tier:
    • Fluent Bit: per-node memory, error rates, backoff/retries.
    • Kafka: broker CPU, disk usage, partition lag, consumer lag.
    • Stream processors: processing rate, error queues, retry behavior.
    • Elasticsearch: indexing rate, JVM memory, search latency, node availability.
  • Implement backpressure and retry strategies in Fluent Bit and processors.
  • Use topic partitioning and meaningful keys in Kafka to keep related logs grouped (e.g., by service).
  • Keep explicit schema or mappings for hot indices to avoid costly dynamic mapping updates.
  • Implement retention lifecycle policies to automate rollover from hot to cold.
Quick troubleshooting checklist for a noisy-service incident
  • Check Kafka topic lag: are consumers falling behind?
  • Inspect Fluent Bit errors and local buffer usage on nodes.
  • Verify stream processor health and retry queues.
  • Confirm Elasticsearch cluster health and threadpool rejections.
  • If indexing is overwhelmed, stop indexing to hot tier temporarily and rely on Kafka + S3 for durability until capacity is restored.
References and further reading Summary This pipeline — Fluent Bit (DaemonSet) → Kafka → stream processor → Elasticsearch (hot) + S3 (cold) — provides resiliency, scalability, and cost-effectiveness for centralized logging across 100+ microservices. It keeps on-call workflows fast, preserves logs durably, and prevents a single noisy service from taking down the entire logging stack.

Watch Video