Skip to main content
Bob’s team opens Kibana every morning to inspect application logs in Discover. Here’s a typical Saturday-night symptom in that stack:
Late on Saturday, Elasticsearch starts running short of memory. Fluentd keeps forwarding log lines from every node, but Elasticsearch gets slower and slower at accepting them. By Sunday it’s barely accepting any new lines. There are no pod crashes — nothing turns red in the cluster.
An infographic titled "Monday Morning" showing three cartoon server/robot icons on platforms—two blue and smiling, one orange and alarmed—with a small stack of "USER1" items above them. A tall orange gradient bar labeled "Memory" appears on the right.
On Monday morning a teammate asks Bob: why does Discover show nothing since Saturday? Bob, the cluster operator, only hears about the outage from someone walking over — not from the system itself. Why didn’t the logs warn him? The logging pipeline was sick. Any warning generated by Elasticsearch would have had to travel through Fluentd into the same Elasticsearch that was refusing new lines. That’s like a smoke alarm wired to the same fuse as the fire.
Log-based alerts can fail when the logging pipeline itself is impaired. Never rely solely on logs to detect logging-system failures.
What Bob needs is an independent, external signal that watches the stack from the outside: metrics. Metrics are numbers recorded over time — single measurements taken repeatedly so you can track their rise and fall. Instead of sentences in a log, metrics are simple numeric samples with timestamps. Sub-second examples:
  • current memory usage in bytes
  • CPU percentage
  • number of open file descriptors
A single metric repeated every 10 seconds creates a time series you can visualize and alert on.
A dark-themed dashboard slide titled "One Number, Over and Over" with the Elasticsearch logo on the left. It shows a vertical memory usage bar (partially filled) and a small line chart on the right labeled "Memory used."
Elasticsearch knows how much memory it uses, but it doesn’t publish that number in the simple text format monitoring tools expect. Enter the exporter: a lightweight helper process that queries Elasticsearch for internal stats and exposes them on an HTTP endpoint in Prometheus’s exposition format — one numeric metric per line. Exporters commonly listen on a port such as 9114. Example exporter output:
An exporter converts application internals into Prometheus’s exposition format so Prometheus can scrape them. Exporters are lightweight and typically run alongside the application they monitor. See Prometheus exporters: https://prometheus.io/docs/instrumenting/exporters/
Fluentd and Kibana also expose metrics via plugins, but Elasticsearch is a good starting point because its failure can stop the logging pipeline. Who reads the page of numbers? Prometheus — a pull-based monitoring system that periodically visits (scrapes) configured endpoints and stores the sampled metrics. Each scraped page is a Prometheus target; scrape intervals and targets are configurable.
A diagram titled "Scraping Targets" showing two service targets (localhost:9114 and localhost:9090) being scraped by Prometheus, which then stores the data into a history/storage cylinder.
Prometheus accumulates thousands of numeric samples. Humans don’t read raw numbers — they read trends. Grafana connects to Prometheus and renders metrics as charts and dashboards. Elasticsearch memory becomes a time series: it climbs during the day and falls at night. Place multiple panels together and you get a dashboard that reveals system health at a glance.
A presentation slide showing the Prometheus logo on the left with an arrow to the Grafana logo below, and a Grafana chart on the right titled "Elasticsearch memory" with a rising blue line.
Dashboards are great for situational awareness, but they require someone to look. To notify Bob automatically, define alerting rules in Prometheus and evaluate them on every scrape. A simple alerting rule concept:
  • Condition: Elasticsearch heap used / heap max > 90%
  • Duration: for 5 minutes
  • Outcome: raise an alert (transition from pending to firing)
Why a 5-minute window? Short spikes are common; using a duration filters out transient noise. While the condition is true but the time window hasn’t elapsed, the alert is “pending.” After the threshold duration it becomes “firing.” This expression + threshold + duration form an alert rule. Example PromQL-style condition:
A monitoring dashboard showing a Prometheus alert marked "Firing" and a line chart of heap used ÷ total that rises above a 90% threshold. A shaded 5-minute window on the right highlights the period where the metric stays over the threshold.
Prometheus detects a firing alert but does not deliver notifications to people. Delivery is handled by Alertmanager. Alertmanager receives firing alerts from Prometheus, groups and deduplicates them, and forwards notifications to configured receivers — Slack, email, PagerDuty, webhooks, etc. End-to-end flow (one metric):
  • Elasticsearch exposes its memory metric via an exporter.
  • Prometheus scrapes and stores the metric.
  • Grafana visualizes the time series on a dashboard.
  • A Prometheus alerting rule evaluates the metric and fires when the rule conditions are met.
  • Alertmanager delivers the notification to Bob’s phone or team channel.
Summary table: core monitoring components and responsibilities Two practical questions remain: where do these programs run in Bob’s environment (pods, sidecars, separate clusters), and what does the experience look like the first time an alert fires and reaches Bob? We’ll explore deployment patterns and a real alert walkthrough next.

Watch Video