Practical guidance to tune Cilium for improved throughput and latency by using netdev datapath, host routing bypass, and BIG TCP with deployment and testing considerations
In this lesson we cover practical Cilium performance enhancements you can enable to improve throughput and latency. These knobs are commonly used in production and are useful for understanding how Cilium forwards packets at scale. The explanations remain tied to the accompanying diagrams so you can visualize how traffic flows through the host and pod network stacks.Architecture reminder
Each Kubernetes node has a host network namespace (with the main interface to the outside world).
Each Pod runs in its own network namespace.
Pods and the host are typically connected via a veth pair; many of the tunables below replace or bypass parts of that path to reduce host-side processing and latency.
Key tuning areas covered:
netdev datapath (replace veth with the AF_XDP-based datapath)
host routing (bypass host iptables/netfilter for selected flows)
BIG TCP (coalesce TCP segments to reduce packet rate)
Netdev datapath (replace veth)
Cilium can replace the veth pair between host and pod with a kernel/AF_XDP-based datapath (netdev). This reduces host-side overhead and gives better throughput and lower latency compared to the traditional veth path because fewer context switches and less software processing are involved.
To enable netdev at install time, set Cilium’s routing mode to native, select the netdev datapath, enable masquerading if you need SNAT, and turn on kube-proxy replacement. Example Helm install:
routingMode=native — use native eBPF-based routing instead of proxy-based routing.
bpf.datapathMode=netdev — use the netdev (AF_XDP) datapath rather than veth.
bpf.masquerade=true — enables eBPF-based masquerading (SNAT) when required.
kubeProxyReplacement=true — replaces kube-proxy with Cilium’s eBPF-based proxy implementation.
Host routing (bypass the host netfilter/iptables path)
By default, pod traffic may traverse the host network stack and be processed by netfilter/iptables. Host routing lets Cilium bypass parts of the host stack for selectable flows, which reduces overhead and improves forwarding efficiency.
To enable host routing–related optimizations, ensure Cilium’s eBPF masquerade and kube-proxy replacement are enabled. A minimal install or upgrade to enable these features:
In many deployments you will combine these settings with the netdev datapath.
Host routing behavior depends on cluster topology and whether nodes are directly routed or behind NAT/load balancers.
BIG TCP (reduce packet processing by coalescing TCP segments)
On high-speed links (100 Gbps+), the sender can generate millions of packets per second for small MTUs. Kernel offloads (GSO/GRO/LRO/TCP segmentation offload) allow coalescing many segments into larger aggregated packets, drastically reducing the packet processing rate and CPU overhead. Cilium supports enabling BIG TCP to allow much larger aggregated TCP segments (for example, up to ~1 MiB), which can reduce pps from millions to tens of thousands for equivalent throughput.BIG TCP requires kernel and NIC driver support for large GSO/GRO and correct hardware offload behavior. When enabled, you can selectively enable BIG TCP for IPv4 and/or IPv6.
BIG TCP requires kernel and NIC driver support. Test carefully on staging before enabling in production. Incorrect assumptions about offloads or driver behavior can cause performance issues or packet corruption.
Example Helm install enabling BIG TCP for both IPv4 and IPv6:
Always test these optimizations (netdev datapath, host routing, and BIG TCP) in a non-production environment first. Hardware and driver incompatibilities or misconfigured offloads can cause degraded performance or packet corruption.
Wrap-up
Using the netdev datapath to replace veth, enabling host routing to bypass the host netfilter stack, and enabling BIG TCP to reduce packet processing are common levers to improve Cilium throughput and latency. Validate kernel and NIC driver compatibility and run staged tests before broad rollout.
