Demo mTLS - KodeKloud

This guide demonstrates how to enable and validate mTLS (mutual TLS) in Cilium using SPIRE (SPIFFE Runtime Environment). It walks through configuring Cilium Helm values for encryption and authentication, installing/upgrading Cilium, deploying SPIRE, deploying a simple server and client, creating a CiliumNetworkPolicy that requires authentication, and observing the authentication flow in the Cilium agent logs. What you’ll learn:

How to enable transparent encryption and SPIRE-based authentication in Cilium.
How to deploy test workloads and apply a policy that enforces mTLS.
How to interpret the agent logs that show the authentication flow.

Prerequisites

Resource	Purpose	Link / Example
Kubernetes cluster	Where Cilium and workloads run	https://kubernetes.io/
Cilium (Helm-managed)	CNI and policy agent	https://cilium.io/
Helm client	Install/upgrade Cilium chart	https://helm.sh/
kubectl	Cluster access and resource operations	https://kubernetes.io/docs/reference/kubectl/overview/
cilium CLI (optional)	Toggle debug and inspect Cilium	https://docs.cilium.io/en/stable/cilium_cli/

Enable encryption and authentication in Cilium Helm values Edit your Cilium values file (values.yaml) to enable transparent encryption and the authentication features. Example excerpts:

# values.yaml (excerpt)
# -- Configure L2 announcements (example section)
l2announcements:
  enabled: true

# -- Enable transparent network encryption.
enableXTSocketFallback: true
encryption:
  enabled: true
  # Encryption method: ipsec or wireguard
  type: ipsec

# -- Enable Non-Default-Deny policies
enableNonDefaultDenyPolicies: true

# Configuration for types of authentication for Cilium (beta)
authentication:
  # Enable authentication processing and garbage collection. When disabled,
  # policy enforcement will still block requests that require authentication
  # but authentication requests will not be processed.
  enabled: true
  queueSize: 1024

# Spire integration (for mTLS)
spire:
  enabled: true

Spire must be reachable from Cilium agents for SPIFFE identity issuance and verification to succeed. Ensure network access, node selectors, tolerations, and resource constraints in your Helm values match your environment.

If you run a hardened cluster, adjust node selectors and tolerations for the Spire server and agents in the Helm values. Confirm the Cilium Helm chart version supports the spire integration for your Cilium release.

Install / upgrade Cilium with the updated values Apply the updated Helm values to Cilium:

helm upgrade cilium cilium/cilium -f values.yaml -n kube-system

Restart the operator and agent pods so the changed configuration takes effect:

kubectl -n kube-system rollout restart deployment cilium-operator
kubectl -n kube-system rollout restart daemonset cilium

Enable debug logging for authentication troubleshooting Enable Cilium debug logging to surface authentication events:

# Using the Cilium CLI (if available)
cilium config set debug true

This updates the Cilium config, which will cause agent pods to restart and pick up debug logging. Verify Spire and Cilium resources are running Confirm that Spire and Cilium components are present and running:

kubectl get pods -A

Representative output (trimmed):

NAMESPACE     NAME                                   READY   STATUS    AGE
cilium-spire  spire-agent-xxxxx                      1/1     Running   16m
cilium-spire  spire-server-0                         2/2     Running   16m
kube-system   cilium-xxxxx                           1/1     Running   16m
kube-system   cilium-envoy-xxxxx                    1/1     Running   16m
kube-system   cilium-operator-xxxxx                 1/1     Running   16m

Check services:

kubectl get svc -A

Representative output:

NAMESPACE     NAME          TYPE        CLUSTER-IP      PORT(S)
cilium-spire  spire-server  ClusterIP   10.96.158.64    8081/TCP
kube-system   cilium-envoy  ClusterIP   None            9964/TCP
default       kubernetes    ClusterIP   10.96.0.1       443/TCP

Deploy a simple server and a client Create an NGINX server and a client (netshoot) to test connectivity and authentication. server-deployment-and-service.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: server
  template:
    metadata:
      labels:
        app: server
    spec:
      containers:
      - name: server
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: server-service
spec:
  selector:
    app: server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

client-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      # optional: schedule on a particular node if desired:
      # nodeName: my-cluster-worker
      containers:
      - name: client
        image: nicolaka/netshoot
        command: ["sleep", "999999"]

Apply the manifests:

kubectl apply -f server-deployment-and-service.yaml
kubectl apply -f client-deployment.yaml

Wait until pods are Ready:

kubectl get pods

Representative pod output:

text
```
NAME                          READY   STATUS    RESTARTS   AGE
client-xxxxx                  1/1     Running   0          12s
server-xxxxx                  1/1     Running   0          8s
```

Test connectivity from client to server
Exec into the client and curl the server service to verify connectivity before mTLS is enforced:

```bash
kubectl exec -it $(kubectl get pod -l app=client -o jsonpath='{.items[0].metadata.name}') -- bash

# inside client shell
curl server-service
```

You should see the NGINX default page, confirming connectivity.

Create a CiliumNetworkPolicy to allow ingress to the server
First create a policy that allows ingress to the server on port 80 (no authentication requirement yet):

policy.yaml

```yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: all-server
spec:
  endpointSelector:
    matchLabels:
      app: server
  ingress:
  - toPorts:
    - ports:
      - port: "80"
        protocol: TCP
```

Apply the policy:

```bash
kubectl apply -f policy.yaml
# Output:
# ciliumnetworkpolicy.cilium.io/all-server created
```

Connectivity should continue to work because the policy currently does not require authentication.

Require authentication (enable mTLS) in the policy
Update the policy to require authentication (mTLS) for the matched traffic:

policy-mtls.yaml

```yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: all-server
spec:
  endpointSelector:
    matchLabels:
      app: server
  ingress:
  - toPorts:
    - ports:
      - port: "80"
        protocol: TCP
    authentication:
      mode: required
```

Apply the updated policy:

```bash
kubectl apply -f policy-mtls.yaml
# Output:
# ciliumnetworkpolicy.cilium.io/all-server configured
```

Behavior on first authenticated connection attempt
When a policy requires authentication, the first packet that triggers authentication will be dropped while Cilium performs the authentication handshake via SPIRE. This is expected: Cilium will initiate the SPIRE-based authentication flow and, once identities are validated, subsequent packets for that connection are allowed. Expect a small delay on the first request.

Observe authentication activity in Cilium agent logs
To trace the authentication flow, tail the Cilium agent logs for the node hosting the server pod.

1. Find the server pod's node:

```bash
kubectl get pod -o wide -l app=server
```

2. Get the Cilium agent pod on that node (namespace kube-system) and tail logs, filtering for authentication messages:

```bash
kubectl -n kube-system -c cilium-agent logs <cilium-agent-pod> --timestamps=true -f | grep -E "Policy is requiring authentication|Validating Server SNI|Validated certificate|Successfully authenticated"
```

Example filtered output (representative):

```text
2025-06-05T01:19:09.316809494Z time=2025-06-05T01:19:09Z level=debug msg="Policy is requiring authentication" module=agent.controlplane.auth key="localIdentity=47107, remoteIdentity=35152, remoteNodeID=8555, authType=spire"
2025-06-05T01:19:09.321705678Z time=2025-06-05T01:19:09Z level=debug msg="Validating Server SNI" module=agent.controlplane.auth SNI_ID=35152
2025-06-05T01:19:09.321748394Z time=2025-06-05T01:19:09Z level=debug msg="Validated certificate" module=agent.controlplane.auth uri-san=[spiffe://spiffe.cilium/identity/35152]
2025-06-05T01:19:09.322644706Z time=2025-06-05T01:19:09Z level=debug msg="Successfully authenticated" module=agent.controlplane.auth key="localIdentity=47107, remoteIdentity=35152, remoteNodeID=8555, authType=spire" remote_node_ip=10.0.0.142
```

Authentication flow summary
- First packet is dropped when the policy requires authentication because the mTLS session has not yet been established.
- Cilium detects the required authentication and initiates a SPIRE-based handshake between source and destination workloads.
- The server SNI (Server Name Indication) is validated, SPIFFE identities are retrieved and validated.
- After validation, the agent logs a successful authentication and allows subsequent packets for that connection.

Checklist to enable mTLS in Cilium
1. Enable transparent encryption (ipsec or WireGuard) in Helm values.
2. Set authentication.enabled: true and enable spire: enabled in values.yaml.
3. Helm upgrade and restart Cilium components so configuration is applied.
4. Deploy workloads and create a CiliumNetworkPolicy that selects the target endpoints.
5. Add authentication.mode: required to the policy to enforce mTLS.
6. Enable debug logging and tail agent logs to validate the authentication handshake.

Links and references
- Cilium: https://cilium.io/
- SPIRE: https://spiffe.io/spire/
- SPIFFE: https://spiffe.io/
- CiliumNetworkPolicy reference: https://docs.cilium.io/en/stable/policy/language/
- cilium CLI: https://docs.cilium.io/en/stable/cilium_cli/

<CardGroup>
<Card title="Watch Video" icon="video" cta="Learn more" href="https://learn.kodekloud.com/user/courses/cilium-certified-associate-cca/module/50bb84d0-61e7-4f73-a51b-7da0e8338438/lesson/92e4dc74-6da6-42bf-8677-ae012ffdc3eb"/>
</CardGroup>