Kubernetes Troubleshooting for Application Developers

Prerequisites

kubectl port forward

In this lesson, you'll learn how to use the kubectl port-forward command—a powerful technique that enables local access to internal Kubernetes cluster resources without exposing them to the internet. This method is ideal for testing, debugging, or accessing services in development and user acceptance testing environments.

Port forwarding is particularly useful when working with ClusterIP services, such as a typical "notes-app-deployment" running in the "uat" (user acceptance testing) namespace. Since ClusterIP services are only accessible within the cluster, port forwarding creates a secure tunnel from your local machine directly to the service.

Viewing Cluster Services

First, you can list all services across namespaces with the following command:

controlplane ~ ➜ k get svc -A
NAMESPACE             NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default               kubernetes                                   ClusterIP   10.96.0.1        <none>        443/TCP                     44m
dev                   nginx-nginx                                  NodePort    10.110.4.112     <none>        80:31000/TCP                43m
ingress-nginx         ingress-nginx-controller                     NodePort    10.110.86.144    <none>        80:32077/TCP,443:31387/TCP    42m
kube-system           kube-dns                                     ClusterIP   10.108.196.245   <none>        53/UDP,53/TCP,9153/TCP       42m
kube-system           metrics-server                               ClusterIP   10.96.0.10       <none>        443/TCP                     43m
kubernetes-dashboard  kubernetes-dashboard-api                     ClusterIP   10.103.170.18    <none>        443/TCP                     42m
kubernetes-dashboard  kubernetes-dashboard-auth                    ClusterIP   10.102.75.66     <none>        8000/TCP                    43m
kubernetes-dashboard  kubernetes-dashboard-home                    NodePort    10.110.56.139    <none>        8001:32151/TCP,8445:30806/TCP 43m
kubernetes-dashboard  kubernetes-dashboard-kong-proxy              ClusterIP   10.99.121.107    <none>        443/TCP                     42m
kubernetes-dashboard  kubernetes-dashboard-metrics-scraper         ClusterIP   10.104.148.150   <none>        8000/TCP                    43m
kubernetes-dashboard  kubernetes-dashboard-web                     ClusterIP   10.102.150.94    <none>        8000/TCP                    43m
monitoring            grafana                                      ClusterIP   10.109.162.199   <none>        8081/TCP                    42m
monitoring            prometheus-alertmanager                      ClusterIP   10.111.237.88    <none>        9093/TCP                    43m
monitoring            prometheus-alertmanager-headless             ClusterIP   None             <none>        9093/TCP                    43m
monitoring            prometheus-kube-state-metrics                 ClusterIP   10.109.19.42     <none>        9100/TCP                    43m
monitoring            prometheus-prometheus-node-exporter          ClusterIP   10.100.83.134    <none>        9100/TCP                    43m
monitoring            prometheus-prometheus-pushgateway            ClusterIP   10.111.84.241    <none>        9091/TCP                    42m
qa                    argo-cd-argocd-applicationset-controller     ClusterIP   10.99.129.210    <none>        5556/TCP,5557/TCP           42m
qa                    argo-cd-argocd-dex-server                    ClusterIP   10.102.199.57    <none>        6379/TCP                    42m
qa                    argo-cd-argocd-repo-server                   ClusterIP   10.100.214.156   <none>        8081/TCP                    42m
staging               vault-agent-injector-svc                     ClusterIP   10.96.0.195      <none>        80/TCP,443/TCP              42m
staging               vault-internal                               ClusterIP   10.97.174.170    <none>        443/TCP                    42m
uat                   notes-app-deployment                         ClusterIP   None             <none>        8200/TCP,8201/TCP           42m
controlplane ~ ➜

Here, the "notes-app-deployment" service in the uat namespace is configured as ClusterIP, making it accessible only from within the cluster.

Creating a Secure Tunnel with Port Forwarding

To securely access the internal service, you can use the kubectl port-forward command. This command establishes a local tunnel where traffic from a specified local port is redirected to the service's port in the cluster.

For example, to forward local port 8000 to the service's port 80 (which then directs traffic to the pod target port 3000), run the following command:

controlplane ~ ➜ k port-forward -n uat svc/notes-app-deployment 8000:80
Forwarding from 127.0.0.1:8000 -> 3000

Once the port forward is active, you can access the notes application locally. For instance, execute:

curl localhost:8000

The response might resemble a simple HTML page:

<!DOCTYPE html>
<html>
<head>
  <title>Note Keeper</title>
</head>
<body>
  <h1>Note Keeper</h1>
  <div style="display: flex;">
    <form action="/update" method="post">
      <input type="number" style="display: none;" name="noteId" value="1">
      <textarea rows="6" cols="30" placeholder="Type Here..." name="noteContent">
Hey guys, add your important notes here.
      </textarea>
      <br>
      <button type="submit">Update</button>
    </form>
    <form action="/delete" method="post">
      <input type="number" style="display: none;" name="noteId" value="1">
      <button type="submit" style="margin: 0 4px;">X</button>
    </form>
  </div>
  <h1>Add Note</h1>
  <form action="/" method="post">
    <input type="number" style="display: none;" name="noteId">
    <textarea rows="6" cols="30" placeholder="Type Here..." name="noteContent"></textarea>
    <br>
    <button type="submit">Add</button>
  </form>
</body>
</html>

Tip

Remember that port forwarding is temporary—it lasts as long as your terminal session remains active.

Verifying Service Configuration

To inspect the service configuration and verify its port settings, retrieve the YAML definition with the following command:

controlplane ~ ➜ k get svc -n uat notes-app-deployment -o yaml

The command outputs a detailed YAML configuration similar to:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"notes-app-deployment","namespace":"uat"},"spec":{"ports":[{"port":80,"targetPort":3000}],"selector":{"app":"notes-app"}}}
  creationTimestamp: "2024-04-25T14:22:29Z"
  name: notes-app-deployment
  namespace: uat
  resourceVersion: "592"
  uid: aa996b82-96a9-4e43-bf72-2fe1c21e777
spec:
  clusterIP: 10.102.16.119
  clusterIPs:
    - 10.102.16.119
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - port: 80
      protocol: TCP
      targetPort: 3000
  selector:
    app: notes-app
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

In this YAML, notice that the service listens on port 80 and forwards the traffic to port 3000 on the pod.

Accessing Other Internal Services

Similarly, you can port-forward other internal services. For example, if you need secure access to the Kubernetes Dashboard (a ClusterIP service), forward a local port to the Dashboard's port as shown:

controlplane ~ ➜ k port-forward -n kubernetes-dashboard svc/kubernetes-dashboard-web 8000:8000

After running the command, open your web browser and navigate to http://localhost:8000 to access the Kubernetes Dashboard securely.

Additional Use: Service Account Tokens

For added flexibility, you can also create service account tokens when needed. This can be accomplished with the following command:

kubectl -n NAMESPACE create-token SERVICE_ACCOUNT

Security Warning

Use service account tokens cautiously—ensure they are stored securely and only used in trusted contexts.

Summary

In summary, the kubectl port-forward command is an essential tool that enables a secure, temporary tunnel from your local machine directly to services or pods within a Kubernetes cluster. This technique is widely used for debugging, testing APIs, or accessing internal services without the need for an external load balancer.

For further reading on Kubernetes concepts and commands, check out these resources:

This concludes our comprehensive explanation of kubectl port-forward and its many practical applications in managing Kubernetes clusters.

Watch Video

Watch video content

Previous
kubectl exec