Before OpenTelemetry Why Standardization Was the Missing Piece
Explains how OpenTelemetry standardized telemetry collection to solve fragmentation and vendor lock-in, offering vendor-neutral instrumentation and a configurable Collector for flexible observability.
In this lesson we set the context for why OpenTelemetry (OTel) emerged and how it addresses long-standing challenges in collecting telemetry—metrics, logs, and traces—across modern distributed systems. As you read, ask: What was missing before OTel, and how does it fix those gaps?Telemetry is the process of recording and transmitting readings from an instrument. Think of weather systems: satellites, ground sensors, and weather stations all send measurements into a central system for analysis and forecasting.
Similarly, software systems emit signals that reveal the system’s operating condition. Telemetry can originate at multiple layers of the stack: application code, libraries, runtime, OS, kernel, and the underlying hardware. For example, a checkout service running on an EC2 instance is a workload composed of application code and libraries that call the OS and kernel and consume hardware resources.
In distributed systems the overall health is the aggregate of signals from these layers. For brevity we’ll call this unit—application, process, or service—a “software system.”How does telemetry leave a software system? Two common collection modes exist:
Push: an agent or SDK inside the system sends telemetry out (for example, via HTTP POST to a collector or vendor endpoint).
Pull: an external scraper polls an exposed endpoint (for example, Prometheus scraping /metrics) to collect metrics.
You can install an agent on a VM or in a container to push telemetry, or configure your application to call an API/SDK directly. Alternatively, an external scraper can poll the application to collect metrics.
Comparison: Push vs Pull
Mode
How it works
Typical tools / examples
Push
Application/agent sends telemetry to a collector or backend
Agents, SDKs, vendor agents (HTTP/gRPC exporters)
Pull
External system scrapes exposed endpoints
Prometheus scraping /metrics, HTTP health checks
Once collected, telemetry is routed to dedicated systems for visualization and analysis: a metrics system (e.g., Prometheus), a logs pipeline and dashboard (e.g., the ELK stack), and a tracing backend (e.g., Jaeger or Zipkin). Each pipeline typically has its own ingestion, storage, and query tools.
Historically, observability developed in a pillar-based way: metrics, traces, and logs often used separate tools, standards, and vendor pipelines. Different teams adopted different instrumentation approaches—OpenTracing, OpenCensus, vendor-specific agents, or homegrown log agents—resulting in many isolated telemetry silos.
This fragmentation led many organizations to consolidate on a single vendor to reduce operational complexity: deploy that vendor’s agent everywhere and send all telemetry to its backend.
While consolidation reduces short-term complexity, it creates vendor lock-in. Changing backends later can require replacing agents, re-instrumenting code, and migrating pipelines—costly and risky at scale.A better approach is a vendor-neutral layer between systems and backends. Each software system outputs metrics, logs, and traces into that neutral layer; from there, data can be routed to any backend—Vendor A, Vendor B, a data lake, or multiple destinations—without re-instrumenting the source.
With a neutral pipeline, changing where telemetry ends up is a configuration change—not a code or agent replacement. Instrument once; control destinations via telemetry middleware configuration.OpenTelemetry provides that vendor-neutral instrumentation and processing layer. OTel SDKs and agents standardize how applications produce metrics, logs, and traces. The OpenTelemetry Collector acts as configurable middleware to receive, process, and export telemetry to one or many backends. As a result, routing telemetry becomes a matter of editing Collector configuration rather than reinstalling agents.
OTel was designed to coexist with existing vendors and tools. The Collector can export to vendor backends, open-source systems, object stores, or multiple destinations simultaneously—making hybrid observability and gradual migration practical.
OTel supports hybrid environments where some services are instrumented with OpenTelemetry and others are not. You can continue accepting vendor-specific telemetry while incrementally adopting OTel, enabling coexistence and staged migration.To summarize the key challenges before standardization:
Fragmentation: multiple tools and standards across metrics, traces, and logs.
Vendor lock-in: switching backends often required replacing agents and re-instrumenting code.
No single standard: vendors and teams shipped their own agents/SDKs.
Limited flexibility: hard to send the same telemetry to multiple destinations.
Hybrid reality: many environments had a mix of instrumented and non-instrumented services.
OpenTelemetry addresses these issues by standardizing APIs and formats for metrics, logs, and traces, and by providing a flexible Collector to process and export telemetry to any backend. This reduces vendor lock-in, supports gradual adoption, and enables exporting the same telemetry to multiple destinations for future-proofing.
OpenTelemetry provides a vendor-neutral instrumentation layer plus a configurable Collector. Instrument once; export anywhere. This enables flexibility, easier migrations, and hybrid coexistence with existing vendor ecosystems.
That concludes this lesson.References and further reading:
