Routing modes: Tunnel (encapsulation) vs Native routing
Tunnel mode (encapsulation)- Pod A sends a packet to Pod B using Pod IPs (src = Pod A IP, dst = Pod B IP).
- Cilium encapsulates the packet; the outer packet uses node IPs (node1 -> node2).
- The physical network only needs to route node IPs; node2 decapsulates and forwards to Pod B.
- Because of encapsulation (e.g., VXLAN), the physical network does not need knowledge of pod CIDRs.
- Pod A sends a packet to Pod B using Pod IPs and no encapsulation is applied.
- The physical network must route pod CIDRs (for example, 192.168.1.0/24 and 192.168.2.0/24).
- One approach is to add static routes on physical routers pointing each pod CIDR to the respective node IP.
- Static routes do not scale (100 nodes → 100 static routes per router), so dynamic routing with BGP is preferred.
| Mode | Encapsulation | Physical network must know pod CIDRs? | Scalability |
|---|---|---|---|
| Tunnel (encapsulation) | Yes | No | High (no router config) |
| Native routing | No | Yes (or dynamic routing) | High with BGP; poor with static routes |
Why use BGP with Cilium?
BGP lets Cilium form peering sessions with physical routers and automatically advertise:- Pod CIDRs (so routers can forward packets to nodes hosting pods).
- Service IPs (ClusterIP, ExternalIP, LoadBalancerIP) when configured.
Enable the BGP control plane in Cilium
To enable the BGP control plane in the Cilium configuration:Cilium BGP CRDs — what to create
Three Kubernetes custom resources are typically involved:| Resource | Purpose | Example |
|---|---|---|
| CiliumBGPClusterConfig | Which nodes run BGP, cluster/local ASN, and peers | Select nodes by label and define bgpInstances |
| CiliumBGPPeerConfig | Peer-specific settings (timers, multihop, families, advertisement selectors) | Referred by peerConfigRef |
| CiliumBGPAdvertisement | Which prefixes to advertise (PodCIDR, Service IPs, routes attributes) | Controls advertisementType and attributes |
Example resources
CiliumBGPClusterConfig- timers.keepAliveTimeSeconds & holdTimeSeconds: govern BGP keepalive and hold intervals.
- ebgpMultihop: TTL for eBGP multi-hop peerings (required when peer is not directly connected).
- families: choose address families (IPv4/IPv6) and safi (e.g., unicast).
- advertisements selector: controls which Advertisement resources a BGPPeer should consult.
If you rely on native routing (no encapsulation), enable PodCIDR advertisements so your physical routers learn how to reach pod networks. Use node labels (nodeSelector) to control which nodes run the BGP control plane and avoid advertising routes from every node if not required.
Be cautious with ebgpMultihop and route attributes. Incorrect multihop or advertisement configuration can cause session flaps or unexpected route leaks. Always validate on a staging router before applying in production.
Troubleshooting with the Cilium CLI
Useful cilium CLI commands and sample outputs. List BGP peers and session state:- Verify node selector labels match your CiliumBGPClusterConfig.
- Check network reachability between peers (TCP/179 and any configured multi-hop TTL).
- Confirm ASNs and peerAddress values are correct.
- Inspect Cilium agent/operator logs for BGP negotiation messages.
- Validate that advertisement selectors (labels) match the CiliumBGPAdvertisement resources.
Summary & next steps
- BGP with Cilium enables scalable native routing for PodCIDRs and service IPs.
- Enable bgpControlPlane in Cilium, label nodes to run BGP, and create CiliumBGPClusterConfig, CiliumBGPPeerConfig, and CiliumBGPAdvertisement resources.
- Use the cilium CLI to inspect peers, learned routes, and advertised prefixes.
- Test in staging before applying to production routers; validate route attributes and policies on both Cilium and your physical routers.