Fault Injection

We previously discussed circuit breaking. Fault injection complements that work by letting you test how your services behave when their dependencies fail or slow down. Imagine fire drills in an office or school: you run a simulated emergency to make sure people know what to do before a real fire happens. In Istio Service Mesh, fault injection is the same idea for microservices — intentionally introduce delays or errors between services to verify resilience, fallbacks, and observability. Why use fault injection?

Test whether your application degrades gracefully under failure.
Verify fallback logic (for example, returning a cached response, a friendly error, or switching to a backup service).
Ensure timeouts and retries are implemented correctly so requests don’t hang indefinitely.
Identify bugs, misconfigurations, or missing error handling before they become live incidents.
Build confidence that services will operate under real-world failures (Netflix’s Chaos Monkey is a well-known example of this approach).

Fault injection is not a standalone resource — it’s configured inside a VirtualService. The fault block sits at the same level as route within an HTTP route entry. Here is a basic VirtualService that injects a delay:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: app-vs
  namespace: frontend
spec:
  hosts:
    - app-svc
  http:
    - fault:
        delay:
          percentage:
            value: 100.0
          fixedDelay: 5s
      route:
        - destination:
            host: app-svc.frontend.svc.cluster.local
            subset: v1
            port:
              number: 80

This configuration injects a 5-second delay for 100% of requests to app-svc. Use such a policy to observe how your application behaves under sustained latency and to validate timeouts, circuit breakers, and user-facing error handling. Istio also supports aborting requests — returning an HTTP or gRPC error — as another fault injection type. Example:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: app-vs
  namespace: frontend
spec:
  hosts:
    - app-svc
  http:
    - fault:
        abort:
          percentage:
            value: 50.0
          httpStatus: 400
      route:
        - destination:
            host: app-svc.frontend.svc.cluster.local
            subset: v1
            port:
              number: 80

In this example, 50% of matching requests receive an HTTP 400 response. The abort fault also supports grpcStatus for gRPC traffic. Fault-injection options — quick reference

Fault Type	Key fields	Description
Delay	`fixedDelay` (e.g., `5s`), `percentage.value`	Injects latency to matching requests (delay duration and percent of traffic).
Abort	`httpStatus`, `grpcStatus`, `percentage.value`	Returns an immediate error response (HTTP or gRPC) for part of the traffic.

Correct and consistent use of percentage values

Istio uses percent values in the range 0.0 to 100.0 (for example, 10% is value: 10.0, not 0.1).

Examples for small-percentage faults (10%) Abort example (10% of requests get HTTP 400):

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: ratings-route
spec:
  hosts:
    - ratings.prod.svc.cluster.local
  http:
    - route:
        - destination:
            host: ratings.prod.svc.cluster.local
            subset: v1
      fault:
        abort:
          percentage:
            value: 10.0
          httpStatus: 400

Delay example (10% of requests have a 5s delay):

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews-route
spec:
  hosts:
    - reviews.prod.svc.cluster.local
  http:
    - match:
        - sourceLabels:
            env: prod
      route:
        - destination:
            host: reviews.prod.svc.cluster.local
            subset: v1
      fault:
        delay:
          percentage:
            value: 10.0
          fixedDelay: 5s

You can combine match conditions (source labels, headers, URIs, etc.) to scope faults to specific callers, namespaces, or environments.

Use fault injection carefully in production. Start with low percentages and short delays, validate application behavior, and monitor closely. Run aggressive scenarios in staging or dedicated chaos environments before widening scope in production.

Compact reference showing both Abort and Delay configurations:

# Abort
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: ratings-route
spec:
  hosts:
    - ratings.prod.svc.cluster.local
  http:
    - route:
        - destination:
            host: ratings.prod.svc.cluster.local
            subset: v1
      fault:
        abort:
          percentage:
            value: 10.0
          httpStatus: 400

# Delay
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews-route
spec:
  hosts:
    - reviews.prod.svc.cluster.local
  http:
    - match:
        - sourceLabels:
            env: prod
      route:
        - destination:
            host: reviews.prod.svc.cluster.local
            subset: v1
      fault:
        delay:
          percentage:
            value: 10.0
          fixedDelay: 5s

Best practices

Start small: low percentages and short durations.
Scope faults with match conditions to avoid broad impact.
Monitor application and platform metrics (latency, error rates, SLOs) while testing.
Validate fallbacks and circuit breakers before increasing fault intensity.
Prefer running fault injection in staging or dedicated chaos environments before production.

Links and References

Istio Documentation — official reference for VirtualService and fault injection.
Istio Service Mesh course — contextual learning resources.
Netflix Chaos Monkey — example of large-scale chaos engineering practices.

Introduction

Installation & Configuration

Traffic Management

Securing Workloads

Troubleshooting

Conclusion

Watch Video