Demo OpenTelemetry Collector k8s

In this guide you’ll deploy an OpenTelemetry Collector into a Kubernetes cluster (using Helm) and instrument a sample application so it sends telemetry (traces, metrics, logs) to the collector running in-cluster. This walkthrough uses a Helm-based Collector deployment (DaemonSet in this example) and shows how to:

fetch and customize the Collector Helm chart values,
enable Kubernetes-specific presets (attributes, kubelet metrics, logs),
expose node-local OTLP endpoints via host ports,
deploy an instrumented application that sends OTLP to the node-local collector,
optionally forward traces to Jaeger for visualization.

This article demonstrates the direct Helm approach only (OpenTelemetry Operator is out of scope).

Environment

For this demo I created a local kind cluster with three nodes (one control-plane and two workers). Verify nodes with:

kubectl get node
NAME                 STATUS      ROLES           AGE   VERSION
kind-control-plane   Ready       control-plane   39s   v1.33.1
kind-worker          Ready       <none>          25s   v1.33.1
kind-worker2         Ready       <none>          25s   v1.33.1

We use Helm to install the OpenTelemetry Collector chart. Add and verify the Helm repo:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo list
# Example output:
# NAME             URL
# open-telemetry   https://open-telemetry.github.io/opentelemetry-helm-charts

This image shows a webpage from OpenTelemetry's documentation about Kubernetes integration, describing components like the OTLP Receiver, Kubernetes Attributes Processor, and Kubeletstats Receiver. It includes a sidebar menu with documentation sections and a dark theme interface.

1 — Prepare a local values.yaml

Fetch the chart’s default values into a local values.yaml so you can edit the Collector configuration (mode, image, presets, config):

helm show values open-telemetry/opentelemetry-collector > values.yaml

Open values.yaml in your editor/IDE and edit the sections you need: mode, image, presets, and config (receivers/processors/exporters/pipelines). I prefer editing locally for clarity and versioning.

The image shows a Visual Studio Code workspace with a terminal open, displaying a Kubernetes configuration script related to OpenTelemetry collector settings.

2 — Presets: enable built-in Kubernetes integrations

The Helm chart exposes “presets” that add common Kubernetes-related components (kubernetesAttributes processor, kubeletstats receiver, host metrics, file log collection, etc.). Enabling a preset also adjusts RBAC, volumes, and mounts automatically. Example minimal preset snippets to include under presets: in values.yaml:

presets:
  logsCollection:
    enabled: true
    includeCollectorLogs: false
    storeCheckpoints: false
    maxRecombineLogSize: 102400

  kubernetesAttributes:
    enabled: true
    extractAllPodLabels: false
    extractAllPodAnnotations: false

  kubeletMetrics:
    enabled: true

  hostMetrics:
    enabled: false

  kubernetesEvents:
    enabled: false

mode: daemonset
image:
  repository: otel/opentelemetry-collector-k8s
  pullPolicy: IfNotPresent

Enabling kubernetesAttributes enriches telemetry with pod, container, and node metadata (pod name, pod UID, node name, labels, etc.), which is highly valuable when troubleshooting or visualizing traces/metrics/logs.

The image shows a webpage from OpenTelemetry documentation, specifically about the Kubernetes Attributes Processor. It includes a table detailing deployment patterns and usability, along with a description of the processor's functionalities.

There are prebuilt receivers such as kubeletstats to collect node/pod/container metrics:

presets:
  kubeletMetrics:
    enabled: true

# Additional kubelet receiver configuration (if needed)
receivers:
  kubeletstats:
    collection_interval: 10s
    auth_type: serviceAccount
    endpoint: '${env:K8S_NODE_NAME}:10250'
    insecure_skip_verify: true
    metric_groups:
      - node
      - pod
      - container

The image displays a webpage from OpenTelemetry documentation, detailing the configuration of Kubernetes receivers for telemetry data collection. It includes sections for "Kubeletstats Receiver" and "Filelog Receiver" with navigation links on the side.

3 — Mode: daemonset vs deployment (choose the right topology)

A Collector can run as a DaemonSet (node-local), Deployment (centralized), or StatefulSet. Choose based on collection needs, latency, and scaling.

Mode	Use Case	Pros	Cons
DaemonSet	Node-local collection (host metrics, node-local OTLP)	Low network hops, per-node collection, useful for node-local instrumentation	More pods to manage; storage for logs on nodes
Deployment	Centralized collection and processing	Easier to scale collectors, centralized buffering	Potential higher latency and network hops, single ingestion point

DaemonSet ensures a collector runs on every node (useful to keep network hops low and to collect host-level metrics). Deployment gives centralized collectors and may be easier to scale for ingestion/backpressure.

For this demo we use mode: daemonset so apps can send OTLP to a collector running on the same node.

4 — Collector configuration (receivers / processors / exporters / pipelines)

Under config: in values.yaml define receivers, processors, exporters, extensions, and service pipelines. A simple example enabling OTLP receivers and a debug exporter:

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch: {}
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25

  exporters:
    debug: {}

  extensions:
    health_check:
      endpoint: 0.0.0.0:13133

  service:
    telemetry:
      metrics:
        readers:
          - pull:
              exporter:
                prometheus:
                  host: 0.0.0.0
                  port: 8888
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [debug]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [debug]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [debug]

Do not remove the health_check extension unless you update the liveness/readiness probes. Removing it can cause the Helm chart’s probes to fail and result in CrashLoopBackOff.

5 — Install the Collector with Helm

Install the customized chart:

helm install otel-collector open-telemetry/opentelemetry-collector -f values.yaml

Helm should report the release is deployed:

NAME: otel-collector
LAST DEPLOYED: ...
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES: ...

Check the DaemonSet and pods:

kubectl get ds
# NAME                                              DESIRED   CURRENT   READY ...
kubectl get pod
# Pods may initially appear as CrashLoopBackOff while images download or if config errors exist.

Troubleshooting CrashLoopBackOff

If Collector pods crash, inspect logs to find configuration errors:

kubectl logs <collector-pod-name>

Common example error when values.yaml contains an invalid key in exporter configuration:

Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
'exporters' error reading configuration for 'debug': decoding failed due to the following error(s):
'' has invalid keys: verbosity

Fix the config in values.yaml and update the release:

helm upgrade otel-collector open-telemetry/opentelemetry-collector -f values.yaml

After the upgrade the DaemonSet should converge to the desired number of ready pods:

kubectl get ds
# otel-collector-opentelemetry-collector-agent   2   2   2   ...
kubectl get pod
# Two collector pods, 1/1 Running each

6 — How the DaemonSet exposes OTLP on node host ports

When using mode: daemonset, the chart config maps container ports to host ports (hostPort) so OTLP traffic sent to http://<node-ip>:4318 or gRPC to <node-ip>:4317 is delivered to the local collector pod. Inspect the DaemonSet YAML and node IPs:

kubectl get ds otel-collector-opentelemetry-collector-agent -o yaml > ds.yaml
kubectl get node -o wide

Look for port mappings similar to:

ports:
  - containerPort: 4317
    hostPort: 4317
    name: otlp
    protocol: TCP
  - containerPort: 4318
    hostPort: 4318
    name: otlp-http
    protocol: TCP

Now OTLP HTTP to http://<node-ip>:4318/v1/traces or OTLP gRPC to <node-ip>:4317 will reach the node-local collector.

7 — Deploy a sample instrumented application

This sample app (sloppy-networks/traces-py) periodically generates a trace and exports it via OTLP HTTP to the collector. The app reads the collector host from an environment variable K8S_NODE_IP which we populate from the node’s IP using a fieldRef. Python tracer configuration excerpt:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import os

print("app starting")
COLLECTOR_HOST = os.getenv('K8S_NODE_IP')
print(COLLECTOR_HOST)

COLLECTOR_URL = f"http://{COLLECTOR_HOST}:4318/v1/traces"

def configure_tracer():
    exporter = OTLPSpanExporter(endpoint=COLLECTOR_URL)
    span_processor = BatchSpanProcessor(exporter)
    resource = Resource.create({
        "service.name": "flight-service"
    })
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(span_processor)
    trace.set_tracer_provider(provider)

Kubernetes Deployment app-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: sloppy-networks/traces-py
        env:
          - name: K8S_NODE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.hostIP

Apply the deployment and check which node the pod lands on:

kubectl apply -f app-deployment.yaml
# deployment.apps/myapp created
kubectl get pod -o wide
# myapp-... 1/1 Running   NODE: kind-worker (for example)

Tail the collector logs on the node where the app landed to confirm telemetry arrives and is enriched with Kubernetes attributes:

kubectl logs <collector-pod-on-that-node> -f
# You will see Resource attributes such as k8s.pod.name, k8s.node.name, k8s.deployment.name, etc.

Those attributes are supplied by the kubernetesAttributes preset enabled earlier.

8 — Forward traces to Jaeger (optional)

To visualize traces, deploy Jaeger and configure the Collector to export traces to Jaeger. Example Jaeger Deployment (all-in-one):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  labels:
    app: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
        - name: jaeger
          image: jaegertracing/all-in-one
          ports:
            - containerPort: 16686  # UI
            - containerPort: 4317   # OTLP gRPC
            - containerPort: 4318   # OTLP HTTP ingest
            - containerPort: 14250  # Jaeger gRPC
            - containerPort: 14268  # Jaeger HTTP

Internal ClusterIP Service for Jaeger:

apiVersion: v1
kind: Service
metadata:
  name: jaeger-internal
  labels:
    app: jaeger
spec:
  selector:
    app: jaeger
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
    - name: otlp-http
      port: 4318
      targetPort: 4318
    - name: jaeger-grpc
      port: 14250
      targetPort: 14250
    - name: jaeger-http
      port: 14268
      targetPort: 14268
  type: ClusterIP

Optional NodePort for accessing Jaeger UI from outside the cluster:

apiVersion: v1
kind: Service
metadata:
  name: jaeger-external
  labels:
    app: jaeger
spec:
  selector:
    app: jaeger
  ports:
    - name: ui
      port: 16686
      targetPort: 16686
      nodePort: 30007
  type: NodePort

Apply Jaeger resources:

kubectl apply -f jaeger.yaml
kubectl get svc
# jaeger-internal shown with ports 4317/4318/14250/14268
# jaeger-external NodePort 30007

Update values.yaml for the Collector to include an exporter pointing to jaeger-internal:4317 (OTLP gRPC) or http://jaeger-internal:4318 (OTLP HTTP). Example config.exporters and pipeline snippet:

config:
  exporters:
    otlp/jaeger:
      endpoint: jaeger-internal:4317
      tls:
        insecure: true
  # ... rest of config ...
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/jaeger, debug]

Upgrade the Helm release so the Collector uses the new exporter:

helm upgrade otel-collector open-telemetry/opentelemetry-collector -f values.yaml

Collector pods will restart with the updated configuration. Confirm pods return to Running:

helm list
kubectl get pod
# Collector pods restart and should be Running

Access the Jaeger UI via NodePort (example: http://<node-ip>:30007) and you should see traces from the instrumented flight-service.

Summary checklist

Added the OpenTelemetry Helm repository and used helm show values to create a local values.yaml.
Enabled presets: kubernetesAttributes, kubeletMetrics, logsCollection to simplify Collector configuration and RBAC.
Deployed the Collector as a DaemonSet (node-local) with hostPort mappings for OTLP (4317/4318).
Deployed an instrumented Python app that reads the node IP via status.hostIP and sends OTLP to http://<node-ip>:4318/v1/traces.
(Optional) Deployed Jaeger and configured the Collector to export traces to a ClusterIP jaeger-internal service, then verified traces via Jaeger UI.

This demonstrates a Helm-based flow for deploying an OpenTelemetry Collector on Kubernetes and configuring applications to send telemetry to a node-local collector. For other deployment models and advanced lifecycle management, consider the OpenTelemetry Operator or managed observability backends.

Links and references

OpenTelemetry Helm charts: https://github.com/open-telemetry/opentelemetry-helm-charts
OpenTelemetry Collector docs: https://opentelemetry.io/docs/collector/
Kubernetes docs: https://kubernetes.io/docs/
Jaeger: https://www.jaegertracing.io/

​Environment

​1 — Prepare a local values.yaml

​2 — Presets: enable built-in Kubernetes integrations

​3 — Mode: daemonset vs deployment (choose the right topology)

​4 — Collector configuration (receivers / processors / exporters / pipelines)

​5 — Install the Collector with Helm

​Troubleshooting CrashLoopBackOff

​6 — How the DaemonSet exposes OTLP on node host ports

​7 — Deploy a sample instrumented application

​8 — Forward traces to Jaeger (optional)

​Summary checklist

​Links and references

Watch Video

Environment

1 — Prepare a local values.yaml

2 — Presets: enable built-in Kubernetes integrations

3 — Mode: daemonset vs deployment (choose the right topology)

4 — Collector configuration (receivers / processors / exporters / pipelines)

5 — Install the Collector with Helm

Troubleshooting CrashLoopBackOff

6 — How the DaemonSet exposes OTLP on node host ports

7 — Deploy a sample instrumented application

8 — Forward traces to Jaeger (optional)

Summary checklist

Links and references