Lab Solution Application Failure

In this lab we walk through troubleshooting a simple two‑tier application (web + MySQL). The web UI must display a green success page when the application stack is healthy. Each namespace (alpha, beta, gamma, delta, epsilon, zeta) contains a different failure scenario. For every case we:

Inspect the application resources (Deployment / Pod / Service).
Reproduce the failure (curl the NodePort where applicable).
Identify the root cause and apply a fix.
Verify the web application returns the green success page.

Tip: To avoid repeating -n <namespace> for every kubectl command, set the default namespace for your current context with kubectl config set-context --current --namespace=<namespace>.

Quick reference — Useful kubectl commands

Use these as you go through each namespace:

# set the default namespace for current context
kubectl config set-context --current --namespace=alpha

# inspect resources
kubectl get pods
kubectl get svc
kubectl describe svc <service-name>
kubectl describe deployment <deployment-name>
kubectl logs <pod-name>

# edit/replace resources
kubectl edit svc <service-name>
kubectl edit deployment <deployment-name>
kubectl replace --force -f /tmp/kubectl-edit-XXXXX.yaml

You may also find these links helpful:

Troubleshooting walkthrough by namespace

Below we keep the original sequence and the exact investigation/fix examples so you can reproduce the steps.

1) Namespace: alpha — DNS / Service name mismatch

Symptom: Web UI loads but shows a red (failed) page. The web app logs show it cannot resolve the MySQL service name. Observed error:

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd;
2003: Can't connect to MySQL server on 'mysql-service:3306' (-2 Name does not resolve)
From webapp-mysql-84fbfc644f-nhfsv!

Investigation (select excerpts):

kubectl get pods -n alpha
# webapp-mysql-84fbfc644f-nhfsv   1/1 Running
kubectl get svc -n alpha
# mysql         ClusterIP   10.43.208.47   <none>    3306/TCP
kubectl describe deployment webapp-mysql -n alpha
# ... Environment: DB_Host: mysql-service ...

Root cause: The web app expects mysql-service but the Service is named mysql. In‑cluster DNS requires the Service name to match the hostname the application uses. Fix applied:

Delete or replace the Service so its name matches the DB host used by the app. Example:

kubectl delete svc mysql -n alpha
kubectl create -f /tmp/kubectl-edit-3970124164.yaml -n alpha
# service/mysql-service created
kubectl get svc -n alpha
# mysql-service  ClusterIP   10.43.208.47   <none>   3306/TCP

Verification:

curl http://localhost:30081
# Page shows green success and DB-connected message:
# Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd;
# From webapp-mysql-84fbfc644f-nhfsv!

2) Namespace: beta — Service `targetPort` wrong (3306 vs 8080)

Symptom: Web app reports connection refused when trying to reach MySQL. Observed:

2003: Can't connect to MySQL server on 'mysql-service:3306' (111 Connection refused)

Investigation:

kubectl config set-context --current --namespace=beta
kubectl get pods
kubectl get svc
kubectl describe svc mysql-service
# Port: 3306/TCP
# TargetPort: 8080/TCP
# Endpoints: 10.42.0.12:8080
kubectl get pods -o wide
# mysql pod IP = 10.42.0.12

Root cause: The Service forwards port 3306 to targetPort 8080 on the pod, but MySQL listens on 3306. Traffic hits the wrong container port and is refused. Fix applied:

Edit the Service so targetPort: 3306 (match the MySQL container port):

# kubectl edit svc mysql-service -n beta
ports:
- port: 3306
  protocol: TCP
  targetPort: 3306

Verification:

kubectl describe svc mysql-service -n beta
# Endpoints: 10.42.0.12:3306
# Web app UI now shows the green success page

3) Namespace: gamma — Service selector did not match Pod labels (no endpoints)

Symptom: Web page times out and the Service has no endpoints. Investigation:

kubectl config set-context --current --namespace=gamma
kubectl get pods
kubectl get svc
kubectl describe svc mysql-service
# Selector: name=mysql
# Endpoints: <none>
kubectl get pods -o wide
# mysql pod IP: 10.42.0.13
kubectl describe pod mysql
# check pod labels that the service selector should match

Root cause: The Service selector did not match the pod labels, so Kubernetes did not populate endpoints for the Service. Without endpoints, the Service cannot forward traffic to pods. Fix applied:

Edit the Service selector to match the pod labels, or update the pod labels to match the Service selector. Example (edit Service):

# kubectl edit svc mysql-service -n gamma
selector:
  name: mysql

Verification:

kubectl describe svc mysql-service -n gamma
# Endpoints: 10.42.0.13:3306
# Web application loads and shows success.

4) Namespace: delta — Incorrect DB user configured in web app

Symptom: Web app reports access denied for user sql-user. Observed:

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=sql-user; DB_Password=paswrd;
1045 (28000): Access denied for user 'sql-user'@'10.42.0.16' (using password: YES)
From webapp-mysql-7fbcc4fb8f-dfmnw!

Investigation:

kubectl config set-context --current --namespace=delta
kubectl get pods
kubectl describe deployment webapp-mysql
# env: DB_Host: mysql-service
# env: DB_User: sql-user
# env: DB_Password: paswrd

Root cause: The application is configured to authenticate as sql-user, which does not exist or lacks required privileges on the MySQL server. The intended credential is root. Fix applied:

Edit the Deployment to set DB_User: root and trigger a rollout so the web pod restarts with correct env vars:

kubectl edit deployment webapp-mysql -n delta
# change environment DB_User to "root"

Verification:

After the Deployment updates and a new pod becomes Ready, refresh the web page — it shows green success.

5) Namespace: epsilon — MySQL root password mismatch

Symptom: Even after DB user is set to root, the web app still gets access denied. Inspecting the MySQL pod shows its MYSQL_ROOT_PASSWORD differs from the web app’s DB_Password. Investigation:

kubectl config set-context --current --namespace=epsilon
kubectl get pods
kubectl describe pod mysql
# Environment:
#   MYSQL_ROOT_PASSWORD: passwoooorrddd  <-- not the expected "paswrd"
kubectl describe deployment webapp-mysql
# env: DB_User: root
#      DB_Password: paswrd

Root cause: The MySQL root password configured in the MySQL Pod does not match the password used by the web application. Fix applied:

Ensure both the MySQL pod (or its managing controller) and the web deployment use the same credential. Update the MySQL configuration to MYSQL_ROOT_PASSWORD=paswrd and recreate/redeploy the MySQL Pod so the new password takes effect.

Example sequence:

kubectl edit pod mysql -n epsilon
# update the env MYSQL_ROOT_PASSWORD to "paswrd"
# If editing the Pod directly is not supported, update the controller and replace the pod:
kubectl replace --force -f /tmp/kubectl-edit-XXXXX.yaml -n epsilon

Verification:

After MySQL restarts with the matching root password and the web app uses DB_User=root and DB_Password=paswrd, the web UI displays the green success page.

Do not store credentials in plain text for production. Use Kubernetes Secrets to manage database credentials and ensure deployments reference the same Secret to avoid drift.

6) Namespace: zeta — NodePort incorrect and credentials mismatch

Symptom: Initially you may see “Bad Gateway” or inability to reach the web app. After fixing NodePort, the web app reports DB access denied. Investigation & fixes:

NodePort mismatch
- Found web-service NodePort was 30088 while the environment expects 30081.
- Fix: Edit the Service to set nodePort: 30081.

kubectl config set-context --current --namespace=zeta
kubectl get svc -n zeta
kubectl edit svc web-service -n zeta
kubectl get svc -n zeta
# web-service NodePort 8080:30081/TCP

DB user and password mismatch
- Web Deployment used DB_User: sql-user. Change it to root.
- MySQL pod had a different MYSQL_ROOT_PASSWORD. Update MySQL to use MYSQL_ROOT_PASSWORD=paswrd and recreate/restart the pod so the password is applied.

Example changes:

# Update web app deployment
kubectl edit deployment webapp-mysql -n zeta
# Update mysql pod env (or the controller) then replace:
kubectl edit pod mysql -n zeta
# set MYSQL_ROOT_PASSWORD: paswrd
kubectl replace --force -f /tmp/kubectl-edit-XXXXX.yaml -n zeta

Verification:

After correcting the NodePort and synchronizing DB username/password, the web application displays the green success page.

Summary table — Namespace issues & fixes

Namespace	Symptom	Root cause	Fix
alpha	Web can’t resolve DB host	Service name `mysql` ≠ app DB_Host `mysql-service`	Rename/recreate Service as `mysql-service`
beta	Connection refused to DB	Service targetPort set to 8080 but MySQL listens on 3306	Set `targetPort: 3306`
gamma	Service has no endpoints	Service selector doesn’t match Pod labels	Update Service selector or Pod labels so they match
delta	Access denied for DB user	App uses `sql-user` (nonexistent)	Set `DB_User: root` (or create the user in DB)
epsilon	Access denied (password mismatch)	MySQL root password differs from app `DB_Password`	Sync `MYSQL_ROOT_PASSWORD` with `DB_Password` (use Secrets)
zeta	Bad gateway then DB access denied	NodePort wrong; DB user/password mismatch	Set correct `nodePort`, set `DB_User: root`, sync MySQL root password

Troubleshooting checklist (order to follow)

DNS / Service name: does the Service name match the DB host used by the app?
Service ports: does Service port and targetPort map to the container port MySQL listens on (3306)?
Service selector: does the selector match the Pod labels so endpoints are populated?
App credentials: do the app’s DB_USER/DB_PASSWORD match credentials on the MySQL server?
Service exposure: if you access the web UI via NodePort, is the NodePort value correct?
Logs: check web app and DB logs for authentication/connection errors; use kubectl logs and kubectl describe to gather context.

Following the steps above helps you systematically isolate DNS/name issues, connectivity/port mismatches, selector/endpoint errors, NodePort exposure problems, and credential mismatches — the common causes in this lab.

Introduction

Core Concepts

Scheduling

Logging & Monitoring

Application Lifecycle Management

Cluster Maintenance

Security

Storage

Networking

Design and Install a Kubernetes Cluster

Install "Kubernetes the kubeadm way"

Troubleshooting

Helm Basics (2025 Updates)

Kustomize Basics (2025 Updates)

Other Topics

Mock Exams

Lab Solution Application Failure

Quick reference — Useful kubectl commands

Troubleshooting walkthrough by namespace

1) Namespace: alpha — DNS / Service name mismatch

2) Namespace: beta — Service `targetPort` wrong (3306 vs 8080)

3) Namespace: gamma — Service selector did not match Pod labels (no endpoints)

4) Namespace: delta — Incorrect DB user configured in web app

5) Namespace: epsilon — MySQL root password mismatch

6) Namespace: zeta — NodePort incorrect and credentials mismatch

Summary table — Namespace issues & fixes

Troubleshooting checklist (order to follow)

Watch Video

​Quick reference — Useful kubectl commands

​Troubleshooting walkthrough by namespace

​1) Namespace: alpha — DNS / Service name mismatch

​2) Namespace: beta — Service targetPort wrong (3306 vs 8080)

​3) Namespace: gamma — Service selector did not match Pod labels (no endpoints)

​4) Namespace: delta — Incorrect DB user configured in web app

​5) Namespace: epsilon — MySQL root password mismatch

​6) Namespace: zeta — NodePort incorrect and credentials mismatch

​Summary table — Namespace issues & fixes

​Troubleshooting checklist (order to follow)

Watch Video

Quick reference — Useful kubectl commands

Troubleshooting walkthrough by namespace

1) Namespace: alpha — DNS / Service name mismatch

2) Namespace: beta — Service `targetPort` wrong (3306 vs 8080)

3) Namespace: gamma — Service selector did not match Pod labels (no endpoints)

4) Namespace: delta — Incorrect DB user configured in web app

5) Namespace: epsilon — MySQL root password mismatch

6) Namespace: zeta — NodePort incorrect and credentials mismatch

Summary table — Namespace issues & fixes

Troubleshooting checklist (order to follow)