Demo Fault Injection

In this lesson you’ll apply two Istio fault-injection scenarios to a simple helloworld app: a fixed delay and an abort (HTTP error). The walkthrough shows how to deploy the sample app, run tests from a pod, and apply VirtualService configurations to observe Envoy-injected faults. Keywords: Istio fault injection, VirtualService, Envoy fault filter, delay, abort, helloworld sample Prerequisites:

A Kubernetes cluster with Istio installed and sidecar injection enabled for your namespace.
kubectl configured to talk to the cluster.

Verify the namespace has Istio sidecar injection enabled

# check namespaces and labels
kubectl get ns --show-labels

Example output:

NAME              STATUS   AGE   LABELS
default           Active   21m   istio-injection=enabled,kubernetes.io/metadata.name=default
istio-system      Active   51s   kubernetes.io/metadata.name=istio-system
kube-node-lease   Active   21m   kubernetes.io/metadata.name=kube-node-lease
kube-public       Active   21m   kubernetes.io/metadata.name=kube-public
kube-system       Active   21m   kubernetes.io/metadata.name=kube-system

Deploy the helloworld sample application

kubectl apply -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/helloworld/helloworld.yaml

Expected response:

service/helloworld created
deployment.apps/helloworld-v1 created
deployment.apps/helloworld-v2 created

Create a test pod (use an image with curl available)

kubectl run test --image=curlimages/curl --restart=Never --command -- sleep 3600
kubectl get pods

Example pod list:

NAME                            READY   STATUS    RESTARTS   AGE
helloworld-v1-7459d7b54b-lqtl6  2/2     Running   0          25s
helloworld-v2-654d97458-7vpz4   2/2     Running   0          25s
test                            1/1     Running   0          5s

Confirm the helloworld service

kubectl get svc

Example output:

NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
helloworld  ClusterIP   10.111.116.180   <none>        5000/TCP       37s
kubernetes  ClusterIP   10.96.0.1        <none>        443/TCP        22m

Smoke test the application from the test pod

kubectl exec -ti test -- curl http://helloworld:5000/hello

Sample responses (the service load-balances between v1 and v2):

Hello version: v2, instance: helloworld-v2-654d97458-7vpz4
Hello version: v1, instance: helloworld-v1-7459d7b54b-lqtl6

At this point the app is functioning normally. Circuit-breaking and connection-level controls are configured in DestinationRule resources (if present) — these are separate from VirtualService fault injection.

Verify there are no existing VirtualServices or DestinationRules in the namespace

kubectl get vs
kubectl get destinationrules.networking.istio.io
# No resources found in default namespace.

Quick reference: common fault types

Fault Type	Description	When to use
Delay	Adds latency to selected requests (e.g. `fixedDelay: 5s`)	Simulate slow downstream services or network latency
Abort	Returns an HTTP error (e.g. `httpStatus: 500`)	Simulate service failures or server errors

Do not apply fault injections in production clusters or against production services unless you have explicit permission and proper safeguards. Fault injection will deliberately break or delay traffic.

Inject a fixed 5s delay for 100% of traffic

Save the following VirtualService as vs-delay.yaml:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: hello-world-vs
spec:
  hosts:
    - helloworld
  http:
    - fault:
        delay:
          percentage:
            value: 100.0
          fixedDelay: 5s
      route:
        - destination:
            host: helloworld
            port:
              number: 5000

Apply and verify:

kubectl apply -f vs-delay.yaml
kubectl get vs

Example output:

virtualservice.networking.istio.io/hello-world-vs configured

NAME             GATEWAYS   HOSTS           AGE
hello-world-vs   []         ["helloworld"]  5s

Test the injected delay:

kubectl exec -ti test -- curl http://helloworld:5000/hello
# (response appears after ~5s)

Expected behavior: every request to helloworld:5000 is delayed by approximately 5 seconds.

Change the fault to an abort (inject HTTP error responses)

Save this VirtualService as vs-abort-500.yaml to abort 100% of requests with HTTP 500:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: hello-world-vs
spec:
  hosts:
    - helloworld
  http:
    - fault:
        abort:
          percentage:
            value: 100.0
          httpStatus: 500
      route:
        - destination:
            host: helloworld
            port:
              number: 5000

Apply and test:

kubectl apply -f vs-abort-500.yaml
kubectl exec -ti test -- curl http://helloworld:5000/hello

You will see an Envoy fault filter message similar to:

fault filter abort

To see the injected status code in the HTTP response headers:

kubectl exec -ti test -- curl --head http://helloworld:5000/hello

Example headers:

HTTP/1.1 500 Internal Server Error
content-length: 18
content-type: text/plain
date: Tue, 15 Apr 2025 15:58:03 GMT
server: envoy

Tip: Change httpStatus to 404, 503, etc., to simulate different server responses.

Inject a 50% abort (random failures)

Save this VirtualService (e.g. vs-abort-50.yaml) to abort 50% of requests with HTTP 404:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: hello-world-vs
spec:
  hosts:
    - helloworld
  http:
    - fault:
        abort:
          percentage:
            value: 50.0
          httpStatus: 404
      route:
        - destination:
            host: helloworld
            port:
              number: 5000

Apply and run multiple requests to observe the distribution:

kubectl apply -f vs-abort-50.yaml

# Run 10 requests and show headers to observe 200 vs 404 responses
kubectl exec -ti test -- /bin/sh -c 'for i in $(seq 1 10); do curl --head http://helloworld.default.svc:5000/hello; echo "---"; done'

Expected result: roughly half of the responses return HTTP/1.1 200 OK and half HTTP/1.1 404 Not Found (randomness and sample size affect exact counts). The same pattern can be used for percentage-based delays.

Conditional fault injection (by header match)

You can scope faults to requests that match particular conditions (for example, a specific header). The Istio docs show an example where a 7s delay is injected only for requests with the end-user header equal to jason:

hosts:
  - ratings
http:
  - fault:
      delay:
        fixedDelay: 7s
        percentage:
          value: 100
    match:
      - headers:
          end-user:
            exact: jason
    route:
      - destination:
          host: ratings
          subset: v1

This injects a 7s delay only for requests where end-user: jason is present.

Fault injection in Istio is configured on VirtualService resources. Circuit-breaking and connection-level controls belong in DestinationRule resources — make sure to use the correct resource for each purpose.

References and next steps

Istio VirtualService reference: https://istio.io/latest/docs/reference/config/networking/virtual-service/
Fault injection task guide: https://istio.io/latest/docs/tasks/traffic-management/fault-injection/
Next steps: configure client-side retries and timeouts in Istio to make your clients resilient to transient faults.

Introduction

Installation & Configuration

Traffic Management

Securing Workloads

Troubleshooting

Conclusion

Demo Fault Injection

Watch Video

Practice Lab