Overview of Hubble observability for Kubernetes networking, covering architecture, deployment, CLI and UI usage, troubleshooting, metrics, and enabling Hubble with Cilium
In this lesson we cover the fundamentals of Hubble — the Cilium observability tool for Kubernetes networking. Hubble uses eBPF to capture and analyze network flows in real time, giving operators and developers rich visibility into service-to-service traffic, HTTP requests, Kafka usage, DNS behavior, and policy-enforced denials.
Hubble operates at node and cluster scope, and can aggregate observability across clusters. It answers critical operational questions such as:
Which services communicate with each other?
How frequently do they communicate?
What does the service dependency graph look like?
Which HTTP endpoints are being called and with what response rates?
Which Kafka topics are services producing to or consuming from?
From a troubleshooting standpoint, Hubble helps you quickly narrow root causes:
Is the failure DNS-related or a service misconfiguration?
Is the issue at the application layer (L7/HTTP) or the network/transport layer (L4/TCP)?
Are packets being dropped or connections reset?
Which policies are blocking traffic?
Hubble can surface recent error events and metrics such as:
DNS resolution failures in the last N minutes.
Interrupted TCP connections or connection timeouts.
Rate of unanswered TCP SYN requests.
4xx/5xx HTTP response rates across services and clusters.
Hubble also exposes SLA-related metrics (p95/p99 latency between requests and responses), identifies poorly performing services, and reveals which connections were blocked by network policies or which DNS names were resolved by specific services.Architecture and componentsBelow is a concise overview of the core Hubble components and their roles.
Component
Role
Typical Deployment
Hubble server
Collects flows and visibility data using eBPF on each node
Runs inside the Cilium agent (no per-node pod)
Hubble relay
Aggregates flows from all Hubble servers and provides a single access point
Deployment (pod) in cluster
Hubble CLI
Local client for querying flows and streaming events
Installed on developer/operator workstation
Hubble UI
Web-based visualization for interactive filtering and dependency graphs
Deployment (pod) in cluster
Hubble UI draws dependency graphs (who talks to whom), highlights denied attempts due to policies, and shows statuses you can also obtain via the CLI.Enabling HubbleEnable Hubble via your Cilium Helm values (values.yaml). Typical settings:
Copy
hubble: # Enable Hubble (true by default). enabled: true # Enable Hubble Relay (aggregates flows from Hubble servers) relay: enabled: true # Whether to enable the Hubble UI. ui: enabled: true
After changing values.yaml, upgrade the Cilium Helm chart and restart operator/agent to apply the new configuration:
You can also inspect Hubble-related pods and services with kubectl:
Copy
kubectl get pod -n kube-system | grep -i hubble# Example output:# hubble-relay-59cc4d545b-dv2vc 1/1 Running 0 21hkubectl get svc -n kube-system# Example output:# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE# hubble-peer ClusterIP 10.96.168.58 <none> 443/TCP 21h# hubble-relay ClusterIP 10.96.219.164 <none> 80/TCP 21h# hubble-ui ClusterIP 10.96.132.144 <none> 80/TCP 21h
Hubble server runs inside the Cilium agent on each node (no separate pod per node). Hubble relay and Hubble UI run as separate deployments/pods and aggregate/visualize node-level flow data.
Installing the Hubble CLIInstall the Hubble CLI on your workstation to query the relay or stream flows directly. The Hubble project provides prebuilt release binaries. Example (Linux):
Copy
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)HUBBLE_ARCH=amd64if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi# Download the tarball and its checksumcurl -L --fail --remote-name-all \ "https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-${HUBBLE_ARCH}.tar.gz" \ "https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum"sha256sum --check "hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum"sudo tar xzvf "hubble-linux-${HUBBLE_ARCH}.tar.gz" -C /usr/local/binrm "hubble-linux-${HUBBLE_ARCH}.tar.gz" "hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum"
Accessing Hubble relay/UI from your workstationA common approach is to port-forward the hubble-relay service (or use the cilium helper). Example:
Avoid exposing Hubble relay/UI to the public internet. Use port-forwarding, a secure tunnel, or an authenticated proxy when accessing Hubble from your workstation.
Using the Hubble CLIThe CLI supports expressive filters for drilling into specific flows. Use --help to view all options. Common examples:Show flows to or from a pod named “green” (both directions):
Copy
hubble observe --pod green
Show flows originating from the “green” pod:
Copy
hubble observe --from-pod green
Show flows destined to the “green” pod:
Copy
hubble observe --to-pod green
Filter by destination port and protocol (e.g., HTTP on port 3000):
Copy
hubble observe --to-pod green --port 3000 --protocol http
Filter by verdict (result of packet handling). Valid verdicts include:
FORWARDED
DROPPED
AUDIT
REDIRECTED
ERROR
TRACED
TRANSLATED
Example:
Copy
hubble observe --to-pod green --port 3000 --protocol http --verdict DROPPED
Using the Hubble UIThe Hubble UI provides a graphical way to explore flows, dependency graphs, and policy denials. The cilium CLI can forward and open the UI for you:
Copy
cilium hubble ui# Opening "http://localhost:12000" in your browser...
The UI supports the same filtering options as the CLI (source/destination pod/service, status, protocol, ports, verdicts, time ranges) and is especially useful for interactive debugging and visualizing service dependencies.SummaryHubble (embedded in Cilium) together with Hubble relay and Hubble UI delivers low-overhead, high-fidelity network observability for Kubernetes clusters. Use the CLI for automation and quick inspections; use the UI for visual exploration, dependency graphs, and troubleshooting policy-related denials.Links and references