> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Demo Circuit Breakers > Guide on configuring Istio circuit breaking using DestinationRule connectionPool and outlierDetection, deploying an echo service, using Fortio to generate load, and inspecting Envoy stats This guide shows how to configure circuit breaking in Istio using a simple echo service and Fortio for load testing. You'll deploy an echo application, create a VirtualService and a DestinationRule (with connection pool limits and outlier detection), generate load to observe Envoy rejecting requests (HTTP 503) when limits are exceeded, and inspect sidecar stats to validate behavior. Always verify the namespace has Istio sidecar injection enabled before you deploy. If you deploy into a namespace without the `istio-injection=enabled` label, your pods won't get sidecars and Istio traffic rules (including circuit breaking) won't apply. ## 1) Verify the namespace is labeled Confirm that the target namespace has `istio-injection=enabled` so that Istio sidecars are injected: ```bash theme={null} root@controlplane ~ ➜ kubectl get ns --show-labels NAME STATUS AGE LABELS default Active 24m istio-injection=enabled,kubernetes.io/metadata.name=default istio-system Active 71s kubernetes.io/metadata.name=istio-system kube-node-lease Active 24m kubernetes.io/metadata.name=kube-node-lease kube-public Active 24m kubernetes.io/metadata.name=kube-public kube-system Active 24m kubernetes.io/metadata.name=kube-system ``` If your target namespace is not labeled, either add the label: ```bash theme={null} kubectl label namespace istio-injection=enabled ``` or create a new labeled namespace: ```bash theme={null} kubectl create namespace kubectl label namespace istio-injection=enabled ``` ## 2) Deploy the echo application and Service Create a Deployment for the echo service (file: `echo_deployment.yaml`) and a ClusterIP Service `echo_svc.yaml`. The Service manifest looks like: ```yaml theme={null} apiVersion: v1 kind: Service metadata: name: echo-server labels: app: echo-server spec: ports: - port: 80 name: http selector: app: echo-server ``` Apply the Deployment and Service: ```bash theme={null} root@controlplane ~ ➜ kubectl apply -f echo_deployment.yaml deployment.apps/echo-server created root@controlplane ~ ➜ kubectl apply -f echo_svc.yaml service/echo-server created root@controlplane ~ ➜ kubectl get pods NAME READY STATUS RESTARTS AGE echo-server-64fb4c5655-vzdh6 2/2 Running 0 67s root@controlplane ~ ➜ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE echo-server ClusterIP 10.104.144.230 80/TCP 4m7s kubernetes ClusterIP 10.96.0.1 443/TCP 30m ``` With the sidecar injected, Istio routing and policies (DestinationRule/VirtualService) will apply to traffic for this Service. ## 3) Deploy Fortio (load-testing client) Fortio is a simple load-testing client used to generate HTTP traffic against the echo service. Deploy Fortio from the Istio samples: ```bash theme={null} root@controlplane ~ ➜ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.25/samples/httpbin/sample-client/fortio-deploy.yaml service/fortio created deployment.apps/fortio-deploy created root@controlplane ~ ➜ kubectl get pods NAME READY STATUS RESTARTS AGE fortio-deploy-689bd5969b-l8z2v 2/2 Running 0 9s echo-server-64fb4c5655-vzdh6 2/2 Running 0 67s ``` Test the echo service from the Fortio pod and extract the `HOSTNAME` field in the response: ```bash theme={null} root@controlplane ~ ➜ kubectl exec fortio-deploy-689bd5969b-l8z2v -c fortio -- /usr/bin/fortio curl -quiet http://echo-server | grep -o '"HOSTNAME":"[^"]*"' HTTP/1.1 200 OK content-type: application/json; charset=utf-8 content-length: 1103 date: Tue, 15 Apr 2025 15:35:04 GMT x-envoy-upstream-service-time: 10 server: envoy "HOSTNAME":"echo-server-64fb4c5655-vzdh6" ``` At this point, no circuit breaker is configured, so requests succeed normally. ## 4) Create a VirtualService (basic routing) Create `vs.yaml` to route traffic for `echo-server` through Istio: ```yaml theme={null} apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: echo-vs spec: hosts: - echo-server http: - route: - destination: host: echo-server port: number: 80 ``` Apply the VirtualService: ```bash theme={null} root@controlplane ~ ➜ kubectl apply -f vs.yaml virtualservice.networking.istio.io/echo-vs created ``` Note: The VirtualService handles routing but does not enforce circuit breaking itself. Circuit breaking behavior comes from DestinationRule trafficPolicy settings. ## 5) Create a DestinationRule to enable circuit breaking Create `dr.yaml` containing connection pool limits and outlier detection to exercise circuit breaking: ```yaml theme={null} apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: echo-dr spec: host: echo-server trafficPolicy: connectionPool: tcp: maxConnections: 1 http: http1MaxPendingRequests: 1 maxRequestsPerConnection: 1 outlierDetection: consecutive5xxErrors: 1 interval: 5s baseEjectionTime: 30s maxEjectionPercent: 100 ``` Apply it: ```bash theme={null} root@controlplane ~ ➜ kubectl apply -f dr.yaml destinationrule.networking.istio.io/echo-dr created root@controlplane ~ ➜ kubectl get vs NAME GATEWAYS HOSTS AGE echo-vs ["echo-server"] 3m36s root@controlplane ~ ➜ kubectl get destinationrules.networking.istio.io NAME HOST AGE echo-dr echo-server 7s ``` ### Quick reference — DestinationRule trafficPolicy fields | Field | Purpose | Example value | | ---------------------------------------------- | ---------------------------------------------------- | ------------- | | `connectionPool.tcp.maxConnections` | Maximum concurrent TCP connections per upstream host | `1` | | `connectionPool.http.http1MaxPendingRequests` | Max queued HTTP requests per upstream connection | `1` | | `connectionPool.http.maxRequestsPerConnection` | Max number of requests on a single connection | `1` | | `outlierDetection.consecutive5xxErrors` | Number of consecutive 5xx responses to eject a host | `1` | | `outlierDetection.interval` | Frequency Envoy checks hosts | `5s` | | `outlierDetection.baseEjectionTime` | Duration to eject a host | `30s` | | `outlierDetection.maxEjectionPercent` | Max percent of hosts Envoy can eject | `100` | These fields control how Envoy accepts or rejects requests and when it ejects unhealthy upstream hosts. ## 6) Load test to trigger the circuit breaker Use Fortio to generate concurrent requests. A single request won't trigger circuit breaking — generate concurrent load. First, a small sanity test: ```bash theme={null} root@controlplane ~ ➜ kubectl exec fortio-deploy-689bd5969b-l8z2v -c fortio -- /usr/bin/fortio load -c 2 -qps 0 -n 20 -loglevel Warning http://echo-server ``` Example summary: ```text theme={null} Code 200 : 19 (95.0 %) Code 503 : 1 (5.0 %) All done 20 calls 7.465 ms avg, 260.3 qps ``` Now run a heavier test designed to exceed `maxConnections: 1`: ```bash theme={null} root@controlplane ~ ➜ kubectl exec fortio-deploy-689bd5969b-l8z2v -c fortio -- /usr/bin/fortio load -c 20 -qps 0 -n 80 -loglevel Warning http://echo-server ``` Example output showing the circuit breaker tripping: ```text theme={null} Code 200 : 7 (8.8 %) Code 503 : 73 (91.2 %) All done 80 calls 10.916 ms avg, 1202.9 qps ``` When connection/pending request limits are reached, Envoy rejects excess requests with HTTP 503. ## 7) Adjust the DestinationRule to allow more load If you need to accept more concurrent requests, update the DestinationRule. Example: increase `maxConnections` to 10 and `maxRequestsPerConnection` to 10: ```yaml theme={null} apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: echo-dr spec: host: echo-server trafficPolicy: connectionPool: tcp: maxConnections: 10 http: http1MaxPendingRequests: 1 maxRequestsPerConnection: 10 outlierDetection: consecutive5xxErrors: 1 interval: 5s baseEjectionTime: 30s maxEjectionPercent: 100 ``` Apply the updated rule: ```bash theme={null} root@controlplane ~ ➜ kubectl apply -f dr.yaml destinationrule.networking.istio.io/echo-dr configured ``` Re-run the same Fortio load test: ```bash theme={null} root@controlplane ~ ➜ kubectl exec fortio-deploy-689bd5969b-l8z2v -c fortio -- /usr/bin/fortio load -c 20 -qps 0 -n 80 -loglevel Warning http://echo-server ``` Updated sample summary: ```text theme={null} Code 200 : 15 (18.8 %) Code 503 : 65 (81.2 %) All done 80 calls 13.418 ms avg, 1131.1 qps ``` Tuning these parameters changes how many requests Envoy accepts before it begins rejecting new requests. ## 8) Inspect Envoy / pilot-agent stats for circuit breaker metrics Query the sidecar proxy (pilot-agent) stats to inspect pending/open/rejected metrics for the cluster. From a pod with the Istio sidecar (e.g., Fortio), run: ```bash theme={null} root@controlplane ~ ➜ kubectl exec fortio-deploy-689bd5969b-l8z2v -c istio-proxy -- pilot-agent request GET stats | grep echo-server | grep pending cluster.outbound|80||echo-server.default.svc.cluster.local;.circuit_breakers.default.remaining_pending: 1 cluster.outbound|80||echo-server.default.svc.cluster.local;.circuit_breakers.default.rq_pending_open: 0 cluster.outbound|80||echo-server.default.svc.cluster.local;.upstream_rq_pending_active: 0 cluster.outbound|80||echo-server.default.svc.cluster.local;.upstream_rq_pending_failure_eject: 0 cluster.outbound|80||echo-server.default.svc.cluster.local;.upstream_rq_pending_overflow: 139 cluster.outbound|80||echo-server.default.svc.cluster.local;.upstream_rq_pending_total: 36 ``` Key metrics to watch: * `.upstream_rq_pending_overflow`: number of requests overflowed (rejected due to queue limits). * `.circuit_breakers.default.remaining_pending`: remaining pending capacity. * `.upstream_rq_pending_failure_eject`: pending requests that caused ejection. These metrics help validate whether pending queues, overflows, and ejection events occurred. Circuit breaking and aggressive connection-pool limits can easily cause availability issues. Only apply production limits after you understand the application's requirements and test thoroughly. For exam purposes, remember: circuit breaking is configured via DestinationRule; focus on `connectionPool` and `outlierDetection` fields. ## 9) Reference and example DestinationRule patterns Istio docs contain many DestinationRule examples. Useful patterns include subset-level trafficPolicy, TCP/keepalive tuning, larger outlier detection thresholds, and TLS settings. * Per-service + subset trafficPolicy example: ```yaml theme={null} apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: bookinfo-ratings spec: host: ratings.prod.svc.cluster.local trafficPolicy: loadBalancer: simple: LEAST_REQUEST subsets: - name: testversion labels: version: v3 trafficPolicy: loadBalancer: simple: ROUND_ROBIN ``` * TCP connection pool + keepalive + timeout: ```yaml theme={null} apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: bookinfo-redis spec: host: myredissrv.prod.svc.cluster.local trafficPolicy: connectionPool: tcp: maxConnections: 100 connectTimeout: 30ms tcpKeepalive: time: 7200s interval: 75s ``` * Larger outlier detection thresholds: ```yaml theme={null} apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: reviews-cb-policy spec: host: reviews.prod.svc.cluster.local trafficPolicy: connectionPool: tcp: maxConnections: 100 http: http2MaxRequests: 1000 maxRequestsPerConnection: 10 outlierDetection: consecutive5xxErrors: 7 interval: 5m baseEjectionTime: 15m ``` * Configuring mutual TLS for a service: ```yaml theme={null} apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: db-mtls spec: host: mydbserver.prod.svc.cluster.local trafficPolicy: tls: mode: MUTUAL clientCertificate: /etc/certs/myclientcert.pem privateKey: /etc/certs/client_private_key.pem caCertificates: /etc/certs/rootcacerts.pem ``` For the full DestinationRule reference and more examples, see the Istio documentation: [https://istio.io/latest/docs/reference/config/networking/destination-rule/](https://istio.io/latest/docs/reference/config/networking/destination-rule/) A screenshot of the Istio website documentation page titled "Circuit Breaking," showing the page header, navigation/menu on the left, and explanatory text about configuring circuit breakers. ## Final notes * For the exam: when you see "circuit breaking," think "DestinationRule" and look for `connectionPool` and `outlierDetection`. * Validate changes with controlled load (Fortio or similar) and inspect sidecar stats using `pilot-agent request GET stats`. * Tune connectionPool and outlierDetection values to match real application capacity to avoid unintentional outages. * Next topics to explore: fault injection, retries, and advanced load-balancing policies. Links and References * Istio DestinationRule docs: [https://istio.io/latest/docs/reference/config/networking/destination-rule/](https://istio.io/latest/docs/reference/config/networking/destination-rule/) * Istio Traffic Management overview: [https://istio.io/latest/docs/concepts/traffic-management/](https://istio.io/latest/docs/concepts/traffic-management/)