> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Hands On Monitoring

> Guide showing how to monitor Elasticsearch with an exporter, Prometheus, Grafana and Alertmanager, demonstrating metric flow, alerting rules and webhook receiver integration.

So where do each of the monitoring programs run?

In Bob's lab it's refreshingly simple: a single machine runs four programs, each listening on its own port and passing metrics along the pipeline. Below we follow the path a metric takes, step by step.

![A diagram titled "Four Programs, One Node" showing a controlplane node. Inside the node are four services with icons and ports: exporter (9114), Prometheus (9090), Grafana (3000), and Alertmanager (9093).](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Monitoring-and-Alerting/Hands-On-Monitoring/four-programs-one-node-monitoring-stack.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=2c0eb6a9674e4a69acd82b3279ea1e88)

Quick reference — components and ports:

| Component | Purpose | Default port / example |
| - | - | - |
| Exporter (Elasticsearch exporter) | Exposes Elasticsearch metrics for Prometheus | `9114` |
| Prometheus | Scrapes exporters, stores metrics, evaluates rules | `9090` |
| Grafana | Visualization and ad-hoc queries (Prometheus as data source) | `3000` |
| Alertmanager | Receives alerts from Prometheus and forwards to receivers | `9093` |
| Custom webhook receiver (example) | Receives Alertmanager webhook notifications | `5001` |
| Elasticsearch | Source of metrics (scraped by exporter) | `9200` |

Step-by-step: how the metric travels

1. Elasticsearch exporter — expose metrics

Bob downloads the Elasticsearch exporter, points it at the local Elasticsearch instance (`http://localhost:9200`), and runs it in the background. The exporter exposes Prometheus-format metrics (including JVM memory used/max) on port 9114.

Example commands:

```bash theme={null}
$ wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.7.0/elasticsearch_exporter-1.7.0.linux-amd64.tar.gz
$ tar xzf elasticsearch_exporter-1.7.0.linux-amd64.tar.gz
$ nohup ./elasticsearch_exporter --es.uri http://localhost:9200 &> exporter.log &
```

When you visit `http://localhost:9114/metrics` you’ll see many metrics. Two important metrics the exporter provides are the JVM memory used and the JVM memory max — these are used later in an alert rule.

2. Prometheus — scrape the exporter

Prometheus scrapes exporters according to jobs defined in its configuration. Bob adds a scrape job for the Elasticsearch exporter to `prometheus.yml` and starts Prometheus.

Relevant excerpt from `prometheus.yml`:

```yaml theme={null}
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "elasticsearch"
    static_configs:
      - targets: ["localhost:9114"]
```

Start Prometheus:

```bash theme={null}
$ nohup ./prometheus --config.file=prometheus.yml &> prometheus.log &
```

Open Prometheus UI at `http://localhost:9090`, then go to Status → Targets. You should see both the Prometheus target and the Elasticsearch exporter target marked UP if everything is running.

![A screenshot of the Prometheus targets/status page showing two monitored endpoints (prometheus and elasticsearch) with green "UP" indicators and their metrics URLs. The top title reads "Both Targets Are Up."](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Monitoring-and-Alerting/Hands-On-Monitoring/prometheus-elasticsearch-targets-up.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=eabcad7796048f18c5ec15616a48e1a1)

3. Grafana — visualize Prometheus metrics

Bob runs Grafana (default UI at `http://localhost:3000`). After logging in (`admin` / `admin` on first run) he adds Prometheus as a data source pointing to `http://localhost:9090`. In Grafana’s Explore you can build PromQL queries from available metrics.

Example query to see active primary shards for the cluster named `docker-cluster`:

```promql theme={null}
elasticsearch_cluster_health_active_primary_shards{cluster="docker-cluster"}
```

A steady flat line usually indicates a stable shard count.

4. Alerting — write a rule in Prometheus

Dashboards are great, but they don’t wake people up. Prometheus rules can evaluate expressions and produce alerts when conditions persist.

Bob writes a rule that computes Elasticsearch heap usage as the ratio of used JVM memory to max JVM memory and fires when that ratio stays above 5% for 2 minutes. (This 5% threshold is intentionally low for demonstration; in production use realistic thresholds, e.g. 90%.)

`rules.yml`:

```yaml theme={null}
groups:
- name: elasticsearch
  rules:
  - alert: HighElasticsearchHeapUsage
    expr: elasticsearch_jvm_memory_used_bytes{job="elasticsearch"} / elasticsearch_jvm_memory_max_bytes{job="elasticsearch"} > 0.05
    for: 2m
    labels:
      severity: "critical"
```

![A dark-themed slide titled "Five Percent, on Purpose" showing a vertical "Heap" gauge partially filled to about 5%. The Prometheus logo appears on the left and a small 5% badge and dotted line mark the fill level.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Monitoring-and-Alerting/Hands-On-Monitoring/prometheus-heap-5-percent-gauge.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=6fb89a866a16ffa42b6afb76473ce56f)

This example uses 5% so the alert will fire immediately for demonstration. In real deployments choose realistic thresholds (for example, 90% heap usage).

Tell Prometheus to load the rules file by referencing it in `prometheus.yml`:

```yaml theme={null}
rule_files:
  - "rules.yml"
```

Prometheus reads `rule_files` at startup. You can either restart Prometheus or request a runtime reload by POSTing to `http://localhost:9090/-/reload`. Example of a simple restart approach:

```bash theme={null}
$ ps aux | grep prometheus
root     23871  0.5  2.4  ... ./prometheus --config.file=prometheus.yml
$ kill 23871
$ nohup ./prometheus --config.file=prometheus.yml &> prometheus.log &
```

On the Prometheus Alerts page the new rule will appear as `Pending` while the `for` window is counting down; if the condition persists it will change to `Firing`.

5. Alertmanager — group and route alerts

Prometheus does not notify external systems itself; it forwards alerts to Alertmanager. Bob runs Alertmanager using a configuration that routes alerts to a `web.hook` receiver which posts to a small HTTP server on port 5001.

Relevant excerpt from `alertmanager.yml`:

```yaml theme={null}
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'web.hook'

receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'
```

Start Alertmanager:

```bash theme={null}
$ nohup ./alertmanager --config.file=alertmanager.yml &> alertmanager.log &
```

Tell Prometheus where to send alerts by adding an `alerting` section to `prometheus.yml` (then restart or reload Prometheus):

```yaml theme={null}
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']
```

After this, fired alerts from Prometheus will appear in Alertmanager and follow its routing logic.

![A diagram titled "The Alert Arrives" showing Prometheus firing an alert to Alertmanager, with a browser address bar at localhost:9093/#/alerts. A separate service box labeled localhost:5001 is shown on the right.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Monitoring-and-Alerting/Hands-On-Monitoring/alert-arrives-prometheus-alertmanager-localhost-9093.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=98d2eefb905f6b55cb4619536ae5e19f)

6. Receiver — do something with the alert

To complete the loop Bob runs a simple webhook receiver (example Python script) that listens on port 5001 and prints received alerts. In production this receiver could forward alerts to Slack, PagerDuty, email, or other incident-management tools.

Example run and output:

```bash theme={null}
$ python3 alert_receiver.py
Starting HTTP server on port 5001

Received alert:
{
  "status": "firing",
  "alerts": [
    {
      "labels": {
        "alertname": "HighElasticsearchHeapUsage",
        "instance": "localhost:9114",
        "job": "elasticsearch",
        "severity": "critical"
      },
      "annotations": {},
      "startsAt": "2023-01-01T12:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://localhost:9090/graph?g0..."
    }
  ],
  "groupLabels": {
    "alertname": "HighElasticsearchHeapUsage"
  },
  "commonLabels": {
    "alertname": "HighElasticsearchHeapUsage",
    "severity": "critical"
  },
  "externalURL": "http://localhost:9093"
}
```

The full path is now complete: a metric left Elasticsearch, passed through the exporter, was scraped by Prometheus, tripped a rule, and travelled through Alertmanager to a receiver visible to Bob.

![A diagram titled "That's the Check" showing a monitoring pipeline with icons for Elasticsearch, an exporter, Prometheus, Alertmanager, a receiver, and a user labeled "Bob." It visually represents data flowing through monitoring and alerting components to a human recipient.](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Monitoring-and-Alerting/Hands-On-Monitoring/monitoring-pipeline-elasticsearch-prometheus-alertmanager-bob.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=cf31334666236f32ebead76e4ba64190)

A few weeks later — iterate on the alert

Fast forward: with a production threshold (e.g. 90%) and more services running, the heap alert may fire intermittently. Old Bob would merely restart Elasticsearch; new Bob asks why it’s happening and how to prevent it. Alerts are a starting point for investigation, not the end.

![A dark-themed infographic titled "A Few Weeks Later" showing happy blue service/process icons on the left and a single distressed process in the center. On the right is a tall "Heap" usage bar marked at 90% with a "firing" alert and a user icon labeled "Bob."](https://mintcdn.com/kodekloud-c4ac6d9a/YjasonRdhEsLdkKY/images/Learn-By-Doing-Deploying-and-Managing-the-EFK-Stack-on-Kubernetes/Monitoring-and-Alerting/Hands-On-Monitoring/a-few-weeks-later-heap90-bob.jpg?fit=max\&auto=format\&n=YjasonRdhEsLdkKY\&q=85\&s=092e5b5dedc8cd8ab75b0010197fa323)

Links and references

* Elasticsearch exporter: [https://github.com/prometheus-community/elasticsearch\_exporter](https://github.com/prometheus-community/elasticsearch_exporter)
* Prometheus download and docs: [https://prometheus.io/download/](https://prometheus.io/download/) and [https://prometheus.io/docs/](https://prometheus.io/docs/)
* Grafana: [https://grafana.com/get](https://grafana.com/get)
* Alertmanager docs: [https://prometheus.io/docs/alerting/latest/alertmanager/](https://prometheus.io/docs/alerting/latest/alertmanager/)

Further reading

* PromQL basics — learn common queries for alerting and dashboards.
* Alertmanager routing and receivers — configure Slack, PagerDuty, email, and webhooks.
* Best practices for thresholds and alert fatigue — choose meaningful severity and aggregation windows.

> [card] [Watch Video](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/cda3436b-891b-4d8d-9508-38b75049d84a/lesson/2b47f242-2d5b-41af-9daf-90d7e1e807b7)

> [card] [Practice Lab](https://learn.kodekloud.com/user/courses/learn-by-doing-deploying-and-managing-the-efk-stack-on-kubernetes/module/cda3436b-891b-4d8d-9508-38b75049d84a/lesson/7cca8659-d594-47ae-a3bd-6bf05542c321)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.