Demo Mirroring

In this lesson we configure traffic mirroring with Istio. Traffic mirroring lets you send a copy of live requests to a secondary subset (for testing, canary analysis, logging, or debugging) while the primary subset continues to serve real user traffic.

Make sure the namespace where you deploy your workloads has Istio sidecar injection enabled. Istio features (including mirroring) require injection to be active for the namespace.

Overview:

Primary goal: route 100% of client traffic to v1 while mirroring copies of those requests to v2.
Components used: Kubernetes Deployments, Service, Istio DestinationRule and VirtualService.
Key Istio concepts: subsets (DestinationRule), mirror and mirrorPercentage (VirtualService).

1. Verify namespace injection

Confirm which namespaces have Istio sidecar injection enabled:

kubectl get ns --show-labels

Sample output:

NAME              STATUS   AGE     LABELS
default           Active   2m38s   istio-injection=enabled,kubernetes.io/metadata.name=default
istio-system      Active   2m6s    kubernetes.io/metadata.name=istio-system
kube-node-lease   Active   2m38s   kubernetes.io/metadata.name=kube-node-lease
kube-public       Active   2m38s   kubernetes.io/metadata.name=kube-public
kube-system       Active   2m38s   kubernetes.io/metadata.name=kube-system

If the target namespace is not labeled with istio-injection=enabled, enable injection before deploying the workloads.

2. Deploy two echo-server versions (v1 and v2)

Create a single YAML containing two Deployments: echo-server-v1 and echo-server-v2. Both share app: echo-server but use different version labels so Istio can create subsets that map to those versions.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
      version: v1
  template:
    metadata:
      labels:
        app: echo-server
        version: v1
    spec:
      containers:
      - name: echo
        image: ealen/echo-server
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
      version: v2
  template:
    metadata:
      labels:
        app: echo-server
        version: v2
    spec:
      containers:
      - name: echo
        image: ealen/echo-server
        ports:
        - containerPort: 80

Apply and verify the pods and labels:

kubectl apply -f echo-deployment.yaml
kubectl get pods --show-labels

Sample output (labels show app=echo-server and version):

NAME                                READY   STATUS    RESTARTS   AGE     LABELS
echo-server-v1-59ff75d58-t4dq6      1/2     Running   0          7s      app=echo-server,pod-template-hash=59ff75d58,version=v1,security.istio.io/tlsMode=is
echo-server-v2-5698db4f99-lm5ss     2/2     Running   0          7s      app=echo-server,pod-template-hash=5698db4f99,version=v2,security.istio.io/tlsMode=is,service.istio.io/canonical-name=echo-server,service.istio.io/canonical-revision=v2

3. Create a Service for the echo servers

Expose both versions through a ClusterIP Service that selects on app: echo-server so client traffic can reach either subset depending on routing rules.

apiVersion: v1
kind: Service
metadata:
  name: echo-server
  labels:
    app: echo-server
spec:
  ports:
    - port: 80
      name: http
  selector:
    app: echo-server

Apply and confirm the service:

kubectl apply -f echo-svc.yaml
kubectl get svc --show-labels

Sample output:

NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE    LABELS
echo-server    ClusterIP   10.98.188.49   <none>        80/TCP     5s     app=echo-server
kubernetes     ClusterIP   10.96.0.1      <none>        443/TCP    4m28s  component=apiserver,provider=kubernetes

4. Tail logs for both versions

Open two terminals (or background one process) and stream logs from each pod to observe how mirrored requests appear in the mirror target. Terminal A (v1):

kubectl logs -f echo-server-v1-59ff75d58-t4dq6

Sample v1 log lines:

{"name":"echo-server","hostname":"echo-server-v1-59ff75d58-t4dq6","msg":"[GET] - /","time":"2025-04-11T16:26:41.095Z"}

Terminal B (v2):

kubectl logs -f echo-server-v2-5698db4f99-lm5ss

Sample v2 log lines:

{"name":"echo-server","hostname":"echo-server-v2-5698db4f99-lm5ss","msg":"[GET] - /","time":"2025-04-11T16:26:39.998Z"}

Keep these logs visible while you generate traffic so you can correlate client requests to backend activity.

5. Create a test pod to generate traffic

Run a small pod you can exec into for curl-based testing:

kubectl run test --image=nginx --restart=Never
kubectl exec -it test -- /bin/bash

From inside the test pod, call the echo service repeatedly and extract the responding pod hostname. The echo server returns JSON with a hostname field; use grep and sed to print it cleanly:

curl -s http://echo-server | grep -o '"hostname":"[^"]*"' | sed 's/"hostname":"\(.*\)"/HOSTNAME: \1/'

Example outputs will indicate which backend served each request:

HOSTNAME: echo-server-v2-5698db4f99-lm5ss
HOSTNAME: echo-server-v1-59ff75d58-t4dq6

Watch the logs from step 4 to see mirrored requests appear in v2 while responses come from v1.

6. Define DestinationRule with subsets

Create an Istio DestinationRule that defines v1 and v2 subsets based on the version pod label. These subsets allow VirtualService routing rules to target specific versions.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: echo-server
spec:
  host: echo-server
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

Apply and verify:

kubectl apply -f dr.yaml
kubectl get destinationrules.networking.istio.io

Sample output:

NAME          HOST          AGE
echo-server   echo-server   3s

7. Create a VirtualService that mirrors traffic

The VirtualService below routes 100% of production traffic to subset v1 and mirrors each request to subset v2. mirrorPercentage.value accepts a float between 0.0 and 100.0 so you can partially mirror traffic if desired.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: echo-server
spec:
  hosts:
    - echo-server
  http:
    - route:
        - destination:
            host: echo-server
            subset: v1
          weight: 100
      mirror:
        host: echo-server
        subset: v2
      mirrorPercentage:
        value: 100.0

Apply the VirtualService:

kubectl apply -f vs.yaml

Tip: to reduce resource usage during testing, set mirrorPercentage.value to a lower number (for example 10.0) to mirror only 10% of requests.

8. Test mirrored traffic

From the test pod, run repeated curl commands and observe that client responses continue to come from v1 while v2 receives mirrored copies (visible in its logs). Repeat the same curl command:

curl -s http://echo-server | grep -o '"hostname":"[^"]*"' | sed 's/"hostname":"\(.*\)"/HOSTNAME: \1/'

Expected client responses (all real responses from v1 when routing is 100% to v1):

HOSTNAME: echo-server-v1-59ff75d58-t4dq6
HOSTNAME: echo-server-v1-59ff75d58-t4dq6
HOSTNAME: echo-server-v1-59ff75d58-t4dq6

Meanwhile, check the v2 logs — you should see similar GET entries for the mirrored requests even though the client never received responses from v2. This confirms mirroring is functioning: real traffic is served by v1, and duplicates are sent to v2 for analysis or testing. Example request log from v1:

{
  "name": "echo-server",
  "hostname": "echo-server-v1-59ff75d58-t4dq6",
  "msg": "Fri, 11 Apr 2025 16:29:16 GMT | [GET] - http://echo-server/",
  "time": "2025-04-11T16:29:18.525Z"
}

Example mirrored request log in v2:

{"name":"echo-server","hostname":"echo-server-v2-5698db4f99-lm5ss","msg":"[GET] - http://echo-server/","time":"2025-04-11T16:26:39.998Z"}

Quick reference table

Resource Type	Purpose	Example / Command
Deployment	Run app versions	`echo-server-v1`, `echo-server-v2`
Service	Expose app internally	`kubectl apply -f echo-svc.yaml`
DestinationRule	Define subsets by labels	See `dr.yaml` (subsets `v1` / `v2`)
VirtualService	Route + mirror traffic	See `vs.yaml` (mirror -> `v2`, route -> `v1`)
Test pod	Generate traffic	`kubectl run test --image=nginx --restart=Never`

Notes and references

Mirroring does not change the client response; it only sends a copy of the request to the mirror target.
Use mirrorPercentage to limit the portion of requests that are mirrored (0.0–100.0).
Ensure labels in DestinationRule.subsets exactly match pod labels (version: v1 / version: v2).
Mirroring is useful for testing new versions without impacting users: logs, tracing, or metrics from the mirror can validate behavior.

If you need the Gateway API or other CRDs present, ensure any required CRDs are installed before testing; for example, follow upstream installation instructions for the Gateway API when required by your environment.

Links and references:

That completes this lesson on Istio traffic mirroring. Try adjusting mirrorPercentage and subsets to experiment with partial mirroring and canary analysis in your cluster.

Introduction

Installation & Configuration

Traffic Management

Securing Workloads

Troubleshooting

Conclusion

1. Verify namespace injection

2. Deploy two echo-server versions (v1 and v2)

3. Create a Service for the echo servers

4. Tail logs for both versions

5. Create a test pod to generate traffic

6. Define DestinationRule with subsets

7. Create a VirtualService that mirrors traffic

8. Test mirrored traffic

Quick reference table

Notes and references

Watch Video

Practice Lab

​1. Verify namespace injection

​2. Deploy two echo-server versions (v1 and v2)

​3. Create a Service for the echo servers

​4. Tail logs for both versions

​5. Create a test pod to generate traffic

​6. Define DestinationRule with subsets

​7. Create a VirtualService that mirrors traffic

​8. Test mirrored traffic

​Quick reference table

​Notes and references

Watch Video

Practice Lab

1. Verify namespace injection

2. Deploy two echo-server versions (v1 and v2)

3. Create a Service for the echo servers

4. Tail logs for both versions

5. Create a test pod to generate traffic

6. Define DestinationRule with subsets

7. Create a VirtualService that mirrors traffic

8. Test mirrored traffic

Quick reference table

Notes and references