Demo Common Problems

Documentation Index

Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt

Use this file to discover all available pages before exploring further.

• Cross-namespace mTLS / sidecar injection mismatch (alpha ← charlie)
• Deployment annotation preventing sidecar injection (beta)
• VirtualService destination/port mismatch (delta → httpbin)
• Gateway / VirtualService misconfiguration for external access (alpha → helloworld)

​

1) Cross-namespace call failing: “connection reset by peer”

• From a pod in namespace charlie, a curl to helloworld.alpha.svc.cluster.local:5000/hello fails with “connection reset by peer”.
• Typical root cause: a global mTLS policy (PeerAuthentication) in STRICT mode while the source pod does not have an Istio sidecar injected.

• Verify namespace labels (to confirm injection is enabled):

kubectl get ns --show-labels

• Inspect cluster PeerAuthentication policies (global policy is commonly in istio-system):

kubectl get peerauthentications.security.istio.io -A
kubectl get peerauthentications.security.istio.io -n istio-system default -o yaml

# PeerAuthentication in istio-system
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

• A PeerAuthentication in STRICT mode enforces mTLS for traffic between sidecars. If the client or server pod has no sidecar, Envoy mTLS handshakes fail and you may see “connection reset by peer”.

istioctl analyze -n charlie

1. Label the namespace if it is not labeled:

kubectl label namespace charlie istio-injection=enabled

2. Recreate the source pod so it gets the sidecar. For example:

kubectl delete -f charlie_curl.yaml
kubectl apply -f charlie_curl.yaml

3. Confirm the pod becomes 2/2 (application + istio-proxy):

kubectl get pods -n charlie
# wait until READY shows 2/2

4. Retry the request from the charlie pod (here the pod name is curl):

kubectl exec -n charlie curl -- curl helloworld.alpha.svc.cluster.local:5000/hello

Hello version: v1, instance: helloworld-v1-7459d7b54b-q4mll

• When you discover PeerAuthentication set to STRICT, ensure both client and server pods have sidecars injected. Use istioctl analyze and kubectl get ns --show-labels early in the troubleshooting flow.

​

2) Deployment has an annotation disabling sidecar injection (beta)

• Pods in namespace beta from the curl Deployment show 1/1 READY, meaning the Istio sidecar was not injected even though the namespace may be labeled.

kubectl get deployments.apps -n beta
kubectl describe deployment -n beta curl

Pod Template:
  Annotations:  sidecar.istio.io/inject: false

kubectl get deployment -n beta curl -o yaml

metadata:
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  ...

• The pod template annotation sidecar.istio.io/inject: "false" prevents automatic sidecar injection for pods created by that Deployment, even if the namespace is labeled for injection.

• Edit the Deployment to remove or set the annotation to "true":

kubectl edit deployment -n beta curl
# Remove or change:
# annotations:
#   sidecar.istio.io/inject: "false"

• After editing, a rolling update will create new pods with the sidecar. Verify the new pods are 2/2:

kubectl get pods -n beta
kubectl describe pod -n beta <new-pod-name>

​

3) 503 Service Unavailable due to VirtualService destination/port mismatch (delta → httpbin)

• From charlie, curl httpbin.delta.svc.cluster.local:8000/get returns HTTP/1.1 503 Service Unavailable. This usually means Envoy cannot find an upstream endpoint matching the VirtualService destination.

kubectl get pod -n delta
kubectl get svc -n delta
kubectl exec -n charlie curl -- curl --head httpbin.delta.svc.cluster.local:8000/get
# -> HTTP/1.1 503 Service Unavailable

kubectl get vs -n delta httpbin-vs -o yaml

spec:
  hosts:
  - httpbin.delta.svc.cluster.local
  http:
  - route:
    - destination:
        host: httpbin.charlie.svc.cluster.local
        port:
          number: 5000

• The VirtualService routes traffic to httpbin.charlie.svc.cluster.local:5000, while the actual Kubernetes Service is httpbin.delta.svc.cluster.local listening on port 8000. Envoy cannot resolve an upstream (host/port mismatch), so it replies 503.

• Edit the VirtualService to point to the correct service and port:

kubectl edit vs -n delta httpbin-vs
# Update to:
# spec:
#   hosts:
#   - httpbin.delta.svc.cluster.local
#   http:
#   - route:
#     - destination:
#         host: httpbin.delta.svc.cluster.local
#         port:
#           number: 8000

kubectl exec -n charlie curl -- curl --head httpbin.delta.svc.cluster.local:8000/get
# -> HTTP/1.1 200 OK

• Always confirm the Kubernetes Service name and port that the VirtualService destination expects. Mismatches between VirtualService destinations and Services are a frequent source of 503s.

​

4) External access via Gateway / VirtualService — selector & routing issues (alpha → helloworld)

• Exposing helloworld in namespace alpha via an Istio Gateway and VirtualService. External curl to the ingress IP with Host header hello.kodekloud.com initially fails (connection refused or 404).

1. Find the external ingress IP (istio-ingressgateway):

kubectl get svc -n istio-system
# Look for istio-ingressgateway LoadBalancer IP or NodePort details

2. Inspect the Gateway resource:

kubectl get gateways.networking.istio.io -n alpha hello-gateway -o yaml

spec:
  selector:
    istio: ingress   # INCORRECT selector
  servers:
  - hosts:
    - hello.kodekloud.com
    port:
      number: 80
      protocol: HTTP

• The Gateway’s selector must match the labels on the ingress gateway Pod/Service (commonly istio: ingressgateway). If the selector is wrong, Envoy in the ingress pods will not load or bind the Gateway config and external connections will fail.

kubectl edit gateways.networking.istio.io -n alpha hello-gateway
# Change selector to:
# spec:
#   selector:
#     istio: ingressgateway

3. Inspect the VirtualService:

kubectl get vs -n alpha helloworld-vs -o yaml

spec:
  hosts:
  - helloworld.alpha.svc.cluster.local
  http:
  - route:
    - destination:
        host: helloworld.alpha.svc.cluster.local
        port:
          number: 5000

• If a VirtualService does not include the external hosts (e.g. hello.kodekloud.com) and is not associated with the Gateway via the gateways: field, the ingress Gateway will not route external requests to the service. That results in 404 responses.

kubectl edit vs -n alpha helloworld-vs
# Add:
# spec:
#   gateways:
#   - hello-gateway
#   hosts:
#   - helloworld.alpha.svc.cluster.local
#   - hello.kodekloud.com
#   http:
#   - route:
#     - destination:
#         host: helloworld.alpha.svc.cluster.local
#         port:
#           number: 5000

curl --header "Host: hello.kodekloud.com" http://<INGRESS_IP>/hello
# -> Hello version: v1  (or v2 depending on routing)

HTTP/1.1 200 OK
server: istio-envoy
date: ...
content-type: text/html; charset=utf-8
x-envoy-upstream-service-time: 93

Hello version: v2, instance: helloworld-v2-654d97458-t6vgc

​

Final tips & troubleshooting checklist

• Always confirm Kubernetes Service names and ports before editing VirtualServices.
• Use istioctl analyze to surface common configuration and injection issues.
• When PeerAuthentication is STRICT, both client and server must have Istio sidecars injected.
• Check for pod-template annotations or Deployment-level annotations that disable injection: sidecar.istio.io/inject.
• For Gateways:
  • Ensure Gateway spec.selector matches the ingress gateway labels (commonly istio: ingressgateway).
  • Ensure the VirtualService lists the Gateway name under gateways: and includes the external hosts entry.
• During exams: label namespaces, edit/patch resources, and recreate pods when necessary. Partial credit may be awarded for correct troubleshooting steps even if a full fix is not completed.

• Istio PeerAuthentication docs
• Istio VirtualService docs
• Istio Gateway docs
• istioctl analyze