Demo Common Problems

This article walks through common Istio troubleshooting scenarios you may encounter on exams or in production. For each scenario you’ll find the symptom, step-by-step investigation, root cause analysis, and the recommended fix. Examples use a cluster with these namespaces: alpha, beta, charlie, delta, and istio-system. Quick navigation:

Scenario 1 — Cross-namespace call failing with connection reset (mTLS / sidecar injection)
Scenario 2 — Pod in an injection-enabled namespace is missing sidecar
Scenario 3 — VirtualService routes to wrong host/port causing 503
Scenario 4 — Gateway / External access issues (selector mismatch and missing gateway/hosts in VirtualService)

Summary checklist (use this as a quick reference when troubleshooting Istio):

Area	Common symptoms	Quick validation
Sidecar injection	Pods 1/1 (missing istio-proxy)	`kubectl get pods -n <ns>` and `kubectl get ns --show-labels`
mTLS / PeerAuthentication	Connection reset / TLS handshake errors	`kubectl get peerauthentications.security.istio.io -n istio-system -o yaml`
VirtualService routing	503 from Envoy	`kubectl get virtualservice -n <ns> -o yaml` and verify service FQDN and port
Gateway / Ingress	Connection refused or 404 from external	`kubectl get gateway -n <ns> -o yaml` and match `spec.selector` to ingress pod labels

References:

Scenario 1 — Cross-namespace call failing with connection reset (mTLS / sidecar injection)

Symptom: A pod in namespace charlie fails to curl helloworld in alpha and receives a connection reset:

# from a control plane shell
kubectl exec -n charlie curl -- curl helloworld.alpha.svc.cluster.local:5000/hello
# output
% Total    % Received % Xferd  Average Speed   Time    Time    Time   Current
                                    Dload   Upload   Total   Spent    Left  Speed
0         0         0     --:--:--   --:--:--   --:--:--     0
curl: (56) Recv failure: Connection reset by peer

Investigation steps

Check pods and services in alpha:

kubectl get pods -n alpha
kubectl get svc -n alpha

Check client pods in charlie:

kubectl get pods -n charlie

Verify namespace labels for sidecar injection:

kubectl get ns --show-labels

Example relevant output:

NAME       STATUS   AGE   LABELS
alpha      Active   26m   istio-injection=enabled,kubernetes.io/metadata.name=alpha
charlie    Active   26m   kubernetes.io/metadata.name=charlie

Inspect global mTLS/PeerAuthentication:

kubectl get peerauthentications.security.istio.io -n istio-system default -o yaml

Example (trimmed) YAML:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

Root cause

A global PeerAuthentication in STRICT mode enforces mutual TLS. The client pod in charlie lacked an Istio sidecar because the charlie namespace was not labeled for injection, so it could not establish mTLS and the connection was reset.

Fix

Use istioctl analyze to identify misconfigurations (recommended):

istioctl analyze -n charlie
# Info [IST0102] (Namespace charlie) The namespace is not enabled for Istio injection...

Label the namespace for injection:

kubectl label namespace charlie istio-injection=enabled

Recreate the client pod so it receives an injected sidecar. Example if you have a manifest charlie_curl.yaml:

kubectl delete -f charlie_curl.yaml
kubectl apply -f charlie_curl.yaml

Confirm the pod is Running with both application and istio-proxy containers:

kubectl get pods -n charlie
# NAME   READY   STATUS
# curl   2/2     Running

Retry the curl:

kubectl exec -n charlie curl -- curl helloworld.alpha.svc.cluster.local:5000/hello
# Hello version: v1, instance: helloworld-v1-...

Always verify both that the namespace has the istio-injection=enabled label and that pods were recreated after labeling. Labeling a namespace does not retroactively inject sidecars into existing pods.

Scenario 2 — Pod in an injection-enabled namespace is missing sidecar

Symptom: In beta, pods show 1/1 containers (no istio-proxy) even though the namespace has istio-injection=enabled. Investigation steps

Get pods and deployments in beta and confirm namespace labels:

kubectl get pods -n beta
kubectl get deployments.apps -n beta
kubectl get ns --show-labels

Inspect the Deployment or Pod for annotations that may disable injection:

kubectl describe deployment -n beta curl

Look for an annotation like:

metadata:
  annotations:
    sidecar.istio.io/inject: "false"

Root cause

The Deployment annotation sidecar.istio.io/inject: "false" explicitly disables sidecar injection for those pods, overriding the namespace-level label.

Fix

Edit the Deployment to remove or change the annotation to allow injection (remove or set it to "true"), then recreate pods (rolling update or delete pods to let them be recreated).

Commands:

kubectl edit deployment -n beta curl
# ensure pods are recreated and become 2/2
kubectl get pods -n beta
# e.g. curl-xxxxx   2/2  Running

Verify the pod contains the istio-proxy:

kubectl describe pod -n beta <pod-name> | sed -n '/Containers:/,$p'
# Should list both application container and istio-proxy

Scenario 3 — VirtualService routes to wrong host/port causing 503 Service Unavailable

Symptom: Requests to httpbin in delta return a 503 from Envoy:

kubectl exec -n charlie curl -- curl -I httpbin.delta.svc.cluster.local:8000/get
# HTTP/1.1 503 Service Unavailable
server: envoy

Investigation steps

Validate the Service and Pods in delta:

kubectl get svc -n delta
kubectl get pod -n delta
kubectl get ns --show-labels

Example service output:

kubectl get svc -n delta
# NAME     TYPE        CLUSTER-IP       PORT(S)
# httpbin  ClusterIP   10.96.153.246    8000/TCP

Inspect the VirtualService for httpbin:

kubectl get virtualservice -n delta httpbin-vs -o yaml

Example problematic VirtualService:

spec:
  hosts:
  - httpbin.delta.svc.cluster.local
  http:
  - route:
    - destination:
        host: httpbin.charlie.svc.cluster.local   # WRONG namespace
        port:
          number: 5000                             # WRONG port

Root cause

The VirtualService routes traffic to httpbin.charlie.svc.cluster.local:5000, but the actual service is httpbin.delta.svc.cluster.local on port 8000. Envoy cannot find a valid upstream, resulting in 503.

Fix

Edit the VirtualService to point to the correct host and port:

kubectl edit virtualservice -n delta httpbin-vs
# change destination.host to httpbin.delta.svc.cluster.local
# change destination.port.number to 8000

Verification:

kubectl exec -n charlie curl -- curl -I httpbin.delta.svc.cluster.local:8000/get
# HTTP/1.1 200 OK

Tip: Always cross-check the Kubernetes Service that backs the workloads and ensure VirtualService destinations match the service FQDN and port.

Scenario 4 — Gateway / External access issues (selector mismatch and missing gateway/hosts in VirtualService)

Symptom: helloworld in alpha is reachable internally but not via the configured Gateway/Ingress. External requests to the ingress IP return connection refused or 404. Investigation steps

Check the Ingress Gateway service for external IP or LoadBalancer:

kubectl get svc -n istio-system
# look for istio-ingressgateway (LoadBalancer); EXTERNAL-IP may be pending or an IP

Inspect the Gateway resource in alpha:

kubectl get gateway -n alpha hello-gateway -o yaml

Problem #1 — Gateway selector mismatch Example incorrect selector:

spec:
  selector:
    istio: ingress    # WRONG if the ingress pods are labeled differently
  servers:
  - hosts:
    - hello.kodekloud.com
    port:
      number: 80
      protocol: HTTP

Confirm labels on the ingress pods to determine the correct selector:

kubectl get pods -n istio-system -l app=istio-ingressgateway --show-labels

Ingress pods may have labels like istio=ingressgateway or istio-ingressgateway. The Gateway.spec.selector must match the ingress pod labels exactly; otherwise the Gateway is not bound to any ingress workload and cannot accept traffic. Fix for selector mismatch

Edit the Gateway to match the actual label on the ingress pods:

kubectl edit gateway -n alpha hello-gateway
# change spec.selector to match ingress pod label, for example:
spec:
  selector:
    istio: ingressgateway

After fixing the selector, a previous Connection refused might change to 404 Not Found because the Gateway is now handled by the ingress but the VirtualService still doesn’t match external host/gateway configuration. Problem #2 — VirtualService missing hosts and gateways for external routing

Inspect the helloworld VirtualService:

kubectl get virtualservice -n alpha helloworld-vs -o yaml

Example VirtualService that only contains the internal host:

spec:
  hosts:
  - helloworld.alpha.svc.cluster.local
  http:
  - route:
    - destination:
        host: helloworld.alpha.svc.cluster.local
        port:
          number: 5000

If the VirtualService does not list the external host (for example hello.kodekloud.com) and does not reference the Gateway (gateways), the Gateway will not route requests targeting the external hostname to this VirtualService, and Envoy will return 404. Fix for VirtualService

Edit the VirtualService to include both gateways and the external host:

kubectl edit virtualservice -n alpha helloworld-vs

Example corrected spec:

spec:
  gateways:
  - hello-gateway
  hosts:
  - helloworld.alpha.svc.cluster.local
  - hello.kodekloud.com
  http:
  - route:
    - destination:
        host: helloworld.alpha.svc.cluster.local
        port:
          number: 5000

Verification flow

Ensure the Gateway spec.selector matches the ingress pod labels.
Ensure the VirtualService includes gateways: and the external host in hosts:.
Curl the ingress IP using the external host in the Host header:

curl --head --header "Host: hello.kodekloud.com" http://<INGRESS-IP>/hello
# or
curl --header "Host: hello.kodekloud.com" http://<INGRESS-IP>/hello
# Expect HTTP/1.1 200 OK and a Hello response body

Example successful response:

HTTP/1.1 200 OK
server: istio-envoy

Hello version: v2, instance: helloworld-v2-...

Notes and exam tips

Use istioctl analyze to surface common issues around injection, Gateways, and VirtualServices.
Troubleshooting priority checklist:
1. Validate Kubernetes Service and Pod status.
2. Confirm namespace labels for sidecar injection.
3. Inspect resource-specific configurations: PeerAuthentication, DestinationRule, AuthorizationPolicy, VirtualService, Gateway, and Service.
Common misconfiguration patterns: selector typos, wrong hostnames, incorrect namespaces, and wrong ports.
For exams: apply the correct resource changes where necessary — partial corrections may still earn partial credit; avoid leaving answers blank.

When editing resources: after changing a namespace label or Deployment annotation related to sidecar injection, you must recreate the pods to get the sidecar injected. Labeling alone doesn’t modify existing pods.

This completes the common troubleshooting scenarios covered in this lesson/article.

Introduction

Installation & Configuration

Traffic Management

Securing Workloads

Troubleshooting

Conclusion

Demo Common Problems

Scenario 1 — Cross-namespace call failing with connection reset (mTLS / sidecar injection)

Scenario 2 — Pod in an injection-enabled namespace is missing sidecar

Scenario 3 — VirtualService routes to wrong host/port causing 503 Service Unavailable

Scenario 4 — Gateway / External access issues (selector mismatch and missing gateway/hosts in VirtualService)

Watch Video

​Scenario 1 — Cross-namespace call failing with connection reset (mTLS / sidecar injection)

​Scenario 2 — Pod in an injection-enabled namespace is missing sidecar

​Scenario 3 — VirtualService routes to wrong host/port causing 503 Service Unavailable

​Scenario 4 — Gateway / External access issues (selector mismatch and missing gateway/hosts in VirtualService)

Watch Video

Scenario 1 — Cross-namespace call failing with connection reset (mTLS / sidecar injection)

Scenario 2 — Pod in an injection-enabled namespace is missing sidecar

Scenario 3 — VirtualService routes to wrong host/port causing 503 Service Unavailable

Scenario 4 — Gateway / External access issues (selector mismatch and missing gateway/hosts in VirtualService)