Prometheus Use Case

In this lesson we cover three common Prometheus production use cases: aggregating metrics across locations, alerting on resource exhaustion, and correlating application metrics to identify performance thresholds. These examples show why Prometheus is widely adopted for monitoring and how teams typically operate it in real environments. Start with a common scenario: multiple data centers across the country plus workloads in a cloud provider (for example, AWS). Your goal is a single, unified view of key metrics from all locations—CPU, memory, request latency, error rates, and more. Prometheus can collect and present this data. Typical deployment patterns include running one Prometheus instance per site or environment, then aggregating metrics with federation or centralizing them via remote_write to a long-term store. Prometheus also includes a built-in expression browser and graphing UI for quick exploration. For richer dashboards, teams commonly pair Prometheus with Grafana.

The image illustrates a network diagram with data centers (West DC, Central DC, East DC) and AWS, connected to a central dashboard displaying various graphs and charts.

All of that data can be displayed on a single dashboard by scraping instrumented services or exporters at each location and aggregating results using one of the approaches below.

Aggregation approach	When to use	Key trade-offs
Per-site Prometheus + federation	Multiple independent sites with occasional cross-site queries	Simpler isolation, lower coupling; federation can be complex for global queries
Centralized remote write (`remote_write`)	Long-term storage (clickhouse, Cortex, Thanos, VictoriaMetrics) and unified queries	Enables high-availability and long retention, but adds external storage/manageability
Hybrid (federation + remote_write)	Mix of local alerting and long-term analytics	Balances local resilience with centralized analytics

Use case 2: Alerting on memory saturation

Imagine repeated outages caused by high memory usage on the MySQL host. The operations team wants alerts (email, Slack, PagerDuty) when memory usage hits 80% so they can intervene before users are affected. Prometheus evaluates alerting rules (PromQL) and emits alerts when conditions are true. Alertmanager receives those alerts and handles deduplication, grouping, inhibition, and delivery to receivers such as email, Slack, or PagerDuty.

The image outlines a use case involving server memory issues, where the operations team wants to be notified via email when memory usage reaches 80% capacity to take proactive measures. Diagrams illustrate a process from server monitoring to email notification.

A straightforward alerting rule for high memory usage:

groups:
- name: node.rules
  rules:
  - alert: HighMemoryUsage
    expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 80
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High memory usage on `{{ $labels.instance }}`"
      description: "Memory usage is above 80% for more than 5 minutes on `{{ $labels.instance }}`."

Prometheus evaluates alerting rules; Alertmanager is responsible for routing and delivering alerts to channels (email, Slack, PagerDuty, SMS). Configure Alertmanager receivers and routing rules to control notification behavior and silence management.

Use case 3: Finding the upload size where latency degrades

You’ve shipped a video upload feature and need to know the file size at which request latency begins to increase noticeably. The recommended approach:

Instrument the application to expose upload metrics:
- Use a histogram for request durations, e.g. upload_request_duration_seconds_bucket.
- Record upload sizes as a gauge or histogram (e.g., upload_size_bytes).
- Attach meaningful labels (endpoint, status_code, instance) when appropriate.
Scrape these metrics with Prometheus and visualize them in Grafana or the Prometheus expression browser.
Use PromQL to correlate average upload size and latency over time.

Useful PromQL examples:

Average upload size over the last 5 minutes (if each upload reports upload_size_bytes as a per-request gauge):

avg_over_time(upload_size_bytes[5m])

95th percentile request duration over the last 5 minutes (based on a histogram named upload_request_duration_seconds_bucket):

histogram_quantile(0.95, sum(rate(upload_request_duration_seconds_bucket[5m])) by (le))

Plot both series on a dashboard to visually identify the file-size range where latency increases. You can also create an alert that fires only when both average size and high latency occur together:

(
  avg_over_time(upload_size_bytes[5m]) > 50 * 1024 * 1024
)
and
(
  histogram_quantile(0.95, sum(rate(upload_request_duration_seconds_bucket[5m])) by (le)) > 2
)

This expression becomes true when the 5-minute average upload size exceeds 50 MiB and the 95th-percentile latency exceeds 2 seconds.

When combining vector results with logical operators (for example and), ensure the vectors have compatible label sets. Use aggregation or vector matching (for example, on(instance)) to align labels so the compound expression behaves as intended.

Putting it all together

Prometheus supports the full workflow for these use cases:

Distributed collection and aggregation: run per-site Prometheus instances and consolidate with federation or remote_write.
Alerting: express rules in PromQL, evaluate them in Prometheus, and manage delivery through Alertmanager.
Instrumentation-driven observability: use histograms for latency and size, counters for counts, and gauges for instantaneous values to correlate metrics and drive data-informed decisions.

Recommended checklist for production readiness:

Instrument services with appropriate metric types (histogram for latency, gauge for sizes, counters for totals).
Scrape application endpoints and exporters reliably (configure scrape intervals and relabeling).
Define and test PromQL queries in the expression browser and dashboards.
Create alerting rules with appropriate for durations to avoid flapping.
Configure Alertmanager routes and receivers for on-call workflows.
Choose aggregation/retention strategy (federation vs remote_write) based on query needs and retention requirements.

Links and references

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Prometheus Use Case

Use case 2: Alerting on memory saturation

Use case 3: Finding the upload size where latency degrades

Putting it all together

Watch Video

​Use case 2: Alerting on memory saturation

​Use case 3: Finding the upload size where latency degrades

​Putting it all together

Watch Video

Use case 2: Alerting on memory saturation

Use case 3: Finding the upload size where latency degrades

Putting it all together