Explains observability concepts, the three pillars logs metrics traces, and how Prometheus fits into metrics-based monitoring for troubleshooting and performance in distributed systems.
What is observability?Observability is the ability to infer the internal state of a system from the data it emits. In practice, it means collecting the signals a system produces so you can answer questions about health, performance, and failures—especially in dynamic environments.Observability helps you turn unexpected scenarios into actionable insights. Implementing observability across your applications and infrastructure improves troubleshooting speed, uncovers hard-to-find issues, enables performance monitoring, and enhances cross-team collaboration.
Observability lets you see inside what would otherwise be a black box: without it, data flows in and out but you cannot determine what happened between those points. By exposing internal state and context, you can identify which component failed, why it failed, and what to fix to prevent recurrence.Architectures are getting more complex—particularly with microservices. In a monolithic architecture, state and logs are often centralized; in microservices, many independent components interact, making isolation and diagnosis more difficult.
When troubleshooting, you need more than the symptom—you need the root cause and context. Observability lets you answer questions such as:
Why are error rates increasing?
Why is latency spiking for a subset of requests?
Which service is timing out and why?
These unpredictable questions require the flexibility provided by a complete observability strategy.
How do we accomplish observability?Observability is commonly built on three signal types—often called the three pillars: logs, metrics, and traces. Each pillar answers different questions and is used together for effective monitoring and investigation.
Pillar
What it gives you
Typical examples
Logs
Detailed, timestamped event records for forensic analysis
Aggregated numerical time series for trend analysis and alerting
CPU usage, request latency, error counts (node_filesystem_avail_bytes{fstype="vfat", mountpoint="/home"} 5000)
Traces
Distributed request flows that show timing and causal relationships
End-to-end request traces (trace IDs and spans)
LogsLogs are timestamped records of events with descriptive messages. They are the most common output for applications and infrastructure. Logs provide rich context for specific events but can be very verbose and distributed across services and hosts, which makes manual analysis during outages time-consuming.Example system log snippet:
Oct 26 19:35:00 ub1 kernel: [37510.942568] e1000: enp0s3 NIC Link is DownOct 26 19:35:00 ub1 kernel: [37510.942697] e1000 0000:00:03.0 enp0s3: Reset adapterOct 26 19:35:03 ub1 kernel: [37513.054072] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
TracesTraces follow individual requests as they traverse services and components. A trace is composed of spans; each span records start time, duration, operation name, and parent relationships. Tracing helps pinpoint where latency is added or where failures occur along a request path.
Use traces to:
Visualize end-to-end latency per request.
Identify slow service hops or hotspots.
Correlate errors with specific spans.
MetricsMetrics are numerical measurements sampled over time. They are ideal for identifying trends, triggering alerts, and creating dashboards. Metrics are generally compact and efficient to store and query, which makes them suitable for long-term analysis and automated rule evaluation.A metric typically includes:
Metric name — what is measured.
Value — numeric measurement.
Timestamp — when it was collected.
Labels/dimensions — key/value context for filtering and aggregation.
Where does Prometheus fit?Prometheus is a metrics-first monitoring system: it scrapes numerical time series, evaluates rules, stores metrics efficiently, and triggers alerts based on metric conditions. Prometheus belongs to the metrics pillar and is not a logs or tracing system. For full observability, integrate Prometheus with logging (e.g., Elasticsearch, Logstash, Fluentd) and tracing tools (e.g., Jaeger, Zipkin, OpenTelemetry backends).
Prometheus is built for metrics: it scrapes and stores time series, evaluates alerting rules, and powers dashboards. Use dedicated logging and tracing systems to capture the other observability signals and correlate them with Prometheus metrics.