> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Demo Common Problems > Troubleshooting guide for common Istio issues with diagnostics, root causes, and minimal fixes This article walks through several common Istio troubleshooting scenarios you may encounter in practice or on exams. Each example presents a realistic failure, the diagnostic commands to run, the root cause, and the minimal fix to restore traffic. Contents * Cross-namespace mTLS / sidecar injection mismatch (alpha ← charlie) * Deployment annotation preventing sidecar injection (beta) * VirtualService destination/port mismatch (delta → httpbin) * Gateway / VirtualService misconfiguration for external access (alpha → helloworld) *** ## 1) Cross-namespace call failing: "connection reset by peer" Problem summary * From a pod in namespace `charlie`, a curl to `helloworld.alpha.svc.cluster.local:5000/hello` fails with "connection reset by peer". * Typical root cause: a global mTLS policy (PeerAuthentication) in STRICT mode while the source pod does not have an Istio sidecar injected. Initial checks * Verify namespace labels (to confirm injection is enabled): ```bash theme={null} kubectl get ns --show-labels ``` * Inspect cluster PeerAuthentication policies (global policy is commonly in `istio-system`): ```bash theme={null} kubectl get peerauthentications.security.istio.io -A kubectl get peerauthentications.security.istio.io -n istio-system default -o yaml ``` Example output (abbreviated): ```yaml theme={null} # PeerAuthentication in istio-system apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: default namespace: istio-system spec: mtls: mode: STRICT ``` Diagnosis * A PeerAuthentication in STRICT mode enforces mTLS for traffic between sidecars. If the client or server pod has no sidecar, Envoy mTLS handshakes fail and you may see "connection reset by peer". Use `istioctl analyze` to discover injection-related problems: ```bash theme={null} istioctl analyze -n charlie ``` Fix (enable injection and recreate the pod) 1. Label the namespace if it is not labeled: ```bash theme={null} kubectl label namespace charlie istio-injection=enabled ``` 2. Recreate the source pod so it gets the sidecar. For example: ```bash theme={null} kubectl delete -f charlie_curl.yaml kubectl apply -f charlie_curl.yaml ``` 3. Confirm the pod becomes 2/2 (application + istio-proxy): ```bash theme={null} kubectl get pods -n charlie # wait until READY shows 2/2 ``` 4. Retry the request from the `charlie` pod (here the pod name is `curl`): ```bash theme={null} kubectl exec -n charlie curl -- curl helloworld.alpha.svc.cluster.local:5000/hello ``` Expected response: ```text theme={null} Hello version: v1, instance: helloworld-v1-7459d7b54b-q4mll ``` Best practice * When you discover PeerAuthentication set to STRICT, ensure both client and server pods have sidecars injected. Use `istioctl analyze` and `kubectl get ns --show-labels` early in the troubleshooting flow. If PeerAuthentication is set to `STRICT`, missing sidecars (or non-Istio traffic) will be rejected. Run `istioctl analyze` and check namespace labels (`kubectl get ns --show-labels`) at the start of troubleshooting. *** ## 2) Deployment has an annotation disabling sidecar injection (beta) Problem summary * Pods in namespace `beta` from the `curl` Deployment show `1/1` READY, meaning the Istio sidecar was not injected even though the namespace may be labeled. Confirm the deployment state ```bash theme={null} kubectl get deployments.apps -n beta kubectl describe deployment -n beta curl ``` In the `describe` output you may see: ```text theme={null} Pod Template: Annotations: sidecar.istio.io/inject: false ``` Or view the YAML: ```bash theme={null} kubectl get deployment -n beta curl -o yaml ``` Example snippet: ```yaml theme={null} metadata: annotations: sidecar.istio.io/inject: "false" spec: ... ``` Diagnosis * The pod template annotation `sidecar.istio.io/inject: "false"` prevents automatic sidecar injection for pods created by that Deployment, even if the namespace is labeled for injection. Fix (remove or change the annotation) * Edit the Deployment to remove or set the annotation to `"true"`: ```bash theme={null} kubectl edit deployment -n beta curl # Remove or change: # annotations: # sidecar.istio.io/inject: "false" ``` * After editing, a rolling update will create new pods with the sidecar. Verify the new pods are `2/2`: ```bash theme={null} kubectl get pods -n beta kubectl describe pod -n beta ``` Confirm the `istio-proxy` container is present and `Ready: True` for both containers. On exams and in production, watch for `sidecar.istio.io/inject: "false"` in resource templates. Removing that annotation or setting it to `"true"` will re-enable injection when the namespace is labeled. *** ## 3) 503 Service Unavailable due to VirtualService destination/port mismatch (delta → httpbin) Problem summary * From `charlie`, `curl httpbin.delta.svc.cluster.local:8000/get` returns HTTP/1.1 503 Service Unavailable. This usually means Envoy cannot find an upstream endpoint matching the VirtualService destination. Verify pods, services, and the failing request ```bash theme={null} kubectl get pod -n delta kubectl get svc -n delta kubectl exec -n charlie curl -- curl --head httpbin.delta.svc.cluster.local:8000/get # -> HTTP/1.1 503 Service Unavailable ``` Inspect the VirtualService ```bash theme={null} kubectl get vs -n delta httpbin-vs -o yaml ``` Problematic VirtualService (excerpt): ```yaml theme={null} spec: hosts: - httpbin.delta.svc.cluster.local http: - route: - destination: host: httpbin.charlie.svc.cluster.local port: number: 5000 ``` Diagnosis * The VirtualService routes traffic to `httpbin.charlie.svc.cluster.local:5000`, while the actual Kubernetes Service is `httpbin.delta.svc.cluster.local` listening on port `8000`. Envoy cannot resolve an upstream (host/port mismatch), so it replies 503. Fix (correct the destination host/port) * Edit the VirtualService to point to the correct service and port: ```bash theme={null} kubectl edit vs -n delta httpbin-vs # Update to: # spec: # hosts: # - httpbin.delta.svc.cluster.local # http: # - route: # - destination: # host: httpbin.delta.svc.cluster.local # port: # number: 8000 ``` Verify the fix: ```bash theme={null} kubectl exec -n charlie curl -- curl --head httpbin.delta.svc.cluster.local:8000/get # -> HTTP/1.1 200 OK ``` Best practice * Always confirm the Kubernetes Service name and port that the VirtualService destination expects. Mismatches between VirtualService destinations and Services are a frequent source of 503s. *** ## 4) External access via Gateway / VirtualService — selector & routing issues (alpha → helloworld) Problem summary * Exposing `helloworld` in namespace `alpha` via an Istio Gateway and VirtualService. External curl to the ingress IP with Host header `hello.kodekloud.com` initially fails (connection refused or 404). High-level troubleshooting steps 1. Find the external ingress IP (istio-ingressgateway): ```bash theme={null} kubectl get svc -n istio-system # Look for istio-ingressgateway LoadBalancer IP or NodePort details ``` 2. Inspect the Gateway resource: ```bash theme={null} kubectl get gateways.networking.istio.io -n alpha hello-gateway -o yaml ``` Problematic Gateway (selector mismatch): ```yaml theme={null} spec: selector: istio: ingress # INCORRECT selector servers: - hosts: - hello.kodekloud.com port: number: 80 protocol: HTTP ``` Diagnosis — Gateway selector * The Gateway's `selector` must match the labels on the ingress gateway Pod/Service (commonly `istio: ingressgateway`). If the selector is wrong, Envoy in the ingress pods will not load or bind the Gateway config and external connections will fail. Fix the Gateway selector: ```bash theme={null} kubectl edit gateways.networking.istio.io -n alpha hello-gateway # Change selector to: # spec: # selector: # istio: ingressgateway ``` After fixing the selector you may reach the ingress, but see a 404. This indicates Gateway is bound but the VirtualService is not configuring routes for that Gateway. 3. Inspect the VirtualService: ```bash theme={null} kubectl get vs -n alpha helloworld-vs -o yaml ``` Problematic VirtualService (no gateways and only internal hosts): ```yaml theme={null} spec: hosts: - helloworld.alpha.svc.cluster.local http: - route: - destination: host: helloworld.alpha.svc.cluster.local port: number: 5000 ``` Diagnosis — VirtualService routing for Gateway * If a VirtualService does not include the external `hosts` (e.g. `hello.kodekloud.com`) and is not associated with the Gateway via the `gateways:` field, the ingress Gateway will not route external requests to the service. That results in 404 responses. Fix the VirtualService: add the Gateway and the external host ```bash theme={null} kubectl edit vs -n alpha helloworld-vs # Add: # spec: # gateways: # - hello-gateway # hosts: # - helloworld.alpha.svc.cluster.local # - hello.kodekloud.com # http: # - route: # - destination: # host: helloworld.alpha.svc.cluster.local # port: # number: 5000 ``` Verify success (from outside the cluster; replace ``): ```bash theme={null} curl --header "Host: hello.kodekloud.com" http:///hello # -> Hello version: v1 (or v2 depending on routing) ``` Example HEAD success: ```text theme={null} HTTP/1.1 200 OK server: istio-envoy date: ... content-type: text/html; charset=utf-8 x-envoy-upstream-service-time: 93 ``` And full body: ```text theme={null} Hello version: v2, instance: helloworld-v2-654d97458-t6vgc ``` Common misconfigurations when exposing via Gateway | Issue | What to check | Fix | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | Gateway selector mismatch | Does `spec.selector` match labels on `istio-ingressgateway` service/pods? | Update selector to match `istio: ingressgateway` (or your installation's label). | | VirtualService missing `gateways` or external `hosts` | Does the VirtualService include `gateways:` and the external host (e.g. `hello.kodekloud.com`)? | Add `gateways:` with the Gateway name and include external `hosts`. | | Service/port mismatch | Does the VirtualService's destination port match the Kubernetes Service port? | Ensure VirtualService destination `host` and `port` match the Service. | *** ## Final tips & troubleshooting checklist * Always confirm Kubernetes Service names and ports before editing VirtualServices. * Use `istioctl analyze` to surface common configuration and injection issues. * When PeerAuthentication is STRICT, both client and server must have Istio sidecars injected. * Check for pod-template annotations or Deployment-level annotations that disable injection: `sidecar.istio.io/inject`. * For Gateways: * Ensure Gateway `spec.selector` matches the ingress gateway labels (commonly `istio: ingressgateway`). * Ensure the VirtualService lists the Gateway name under `gateways:` and includes the external `hosts` entry. * During exams: label namespaces, edit/patch resources, and recreate pods when necessary. Partial credit may be awarded for correct troubleshooting steps even if a full fix is not completed. Useful links and references * [Istio PeerAuthentication docs](https://istio.io/latest/docs/reference/config/security/peer_authentication/) * [Istio VirtualService docs](https://istio.io/latest/docs/reference/config/networking/virtual-service/) * [Istio Gateway docs](https://istio.io/latest/docs/reference/config/networking/gateway/) * [istioctl analyze](https://istio.io/latest/docs/ops/diagnostic-tools/istioctl-analyze/) This concludes the demo of common Istio issues and fixes.