Systematic Troubleshooting

All right — welcome back. This lesson covers a practical, repeatable approach to systematic troubleshooting in AWS. You’ll get a troubleshooting mindset, structured frameworks to apply during incidents, and the AWS services and tools that help investigate and resolve faults—from AWS service health to infrastructure, platform, and application-level diagnostics.

Troubleshooting Mindset: Four Core Practices

Adopt these four habits for reliable incident response:

Verify with data: form and validate hypotheses using observable signals.
Isolate variables: make one controlled change at a time to identify cause-and-effect.
Document findings: capture what you tried, what worked, and what didn’t.
Layered analysis: inspect Application → Platform → Infrastructure → AWS Service Health.

A presentation slide titled "The Systematic Troubleshooting Mindset" showing four numbered cards with brief guidance. The cards read: "Verify with Data," "Isolate Variables," "Document Findings," and "Layered Analysis," each with a short explanatory line.

Ask targeted questions and follow the data. Most outages are caused by changes—manual or automated—so documenting incidents and updating runbooks prevents repeat failures.

Causal Analysis: Five Whys

A pragmatic technique for root-cause analysis is the Five Whys: keep asking “Why?” until you reach a systemic cause or a change that introduced the failure. Example chain:

Service unavailable.
Why? Target health checks are failing.
Why? A security group is blocking the health check port.
Why? Security groups were recently updated.
Why? An automated compliance rule changed the configuration.

A horizontal "Five Whys Root Cause Analysis" diagram showing five linked steps: Service unavailable → Target health checks failing → Security group blocks health check port → Recent security group update → Automated compliance rule changed configuration. Each step is represented by a colored circular icon and label.

You may not need all five iterations every time, but this iterative questioning quickly surfaces the change that created the issue.

A Six-Step Troubleshooting Framework

Apply this sequence to structure your triage and remediation:

Define the problem and observable symptoms.
Gather data that confirms the problem (metrics, logs, traces, events).
Form a hypothesis from observed patterns (e.g., memory spike aligns with failure window).
Test the hypothesis (free memory, adjust configuration, or replicate the failure in a sandbox).
Implement a fix and verify return to expected behavior.
Document the incident, timeline, root cause, and remediation steps.

Layered Analysis: Where to Look First

Reason about faults by layer. Typical control and visibility:

Application: full control and instrumentation.
Platform: OS, runtimes, containers, and orchestration (high control).
Infrastructure you manage: EC2, RDS, networking, IAM (partial control).
AWS Service Health: provider-managed; mostly observability.

Check the AWS Service Health Dashboard and the Personal Health Dashboard early in triage to detect provider-side issues. If AWS is implicated, follow any required DR or mitigation plans.

An infographic titled "AWS Service Layer Troubleshooting Model" showing a four-layer colored wedge (01 AWS Service Health, 02 Infrastructure, 03 Platform, 04 Application). To the right is a checklist of infrastructure items to verify, including EC2 instance health, RDS availability, VPC networking, security groups, NACLs, route tables and IAM permissions.

Central Troubleshooting Tools in AWS

CloudWatch: metrics, logs, Logs Insights, alarms, dashboards — use it as your active troubleshooting hub.
X-Ray: distributed tracing and service maps to follow requests across services.
CloudTrail: API-level audit trail for change and access investigations.
AWS Health & Personal Health Dashboard: provider-side events and scheduled maintenance.
Systems Manager (OpsCenter): aggregate signals into OpsItems and associate runbooks for remediation.

A presentation slide titled "Using CloudWatch for Troubleshooting" showing the CloudWatch icon and five feature tiles labeled Metrics, Logs, Logs Insights, Alarms, and Dashboards. Each feature has a colorful circular icon above its label.

Distributed Systems: Tracing with X-Ray

Tracing helps you go from a red node on a service map to the exact trace and subsegment responsible for errors or latency.

A presentation slide titled "AWS X-Ray for Distributed System Troubleshooting" that lists key capabilities. It shows icons and short descriptions for Service Map, Trace Analysis, Segment Details, and Annotations & Metadata.

Change and Configuration: CloudTrail

CloudTrail captures AWS API activity—essential for correlating incidents with configuration changes and identifying actors.

CloudTrail records AWS API calls—not in-guest OS or application-level events. Combine CloudTrail with CloudWatch logs, in-guest logs, and platform-specific diagnostics to build a complete incident timeline.

A slide titled "CloudTrail for Change and Configuration Troubleshooting" showing a laptop with a warning icon and a cloud. Beside it is a checklist of troubleshooting use cases like identifying who modified resources, tracking security group and IAM changes, correlating deployments with incidents, detecting unauthorized API calls, and investigating configuration drift.

AWS Health: Provider-Side Issues

Use:

Personal Health Dashboard for account-specific notices (maintenance, security or operational notifications).
Service Health Dashboard (https://status.aws.amazon.com) for global AWS service status.

An infographic titled "AWS Health for AWS Service Troubleshooting" comparing two panels. The left panel describes the Personal Health Dashboard (scheduled maintenance and security/operational notifications) and the right panel describes the Service Health Dashboard (global AWS service status across regions).

AWS Health events integrate with EventBridge for automation. Route important events into Systems Manager OpsCenter to centralize OpsItems and kick off playbooks (Systems Manager Automation).

A slide titled "Systems Manager OpsCenter for Incident Management" that lists key capabilities—OpsItems, Automatic Aggregation, Runbook Association, Related Resources, and Operational Insights—each with a short description. The slide is branded © KodeKcloud.

Troubleshooting Deployments and CI/CD

When a deployment fails, check the deployment service logs and CloudWatch logs first. Relevant services:

CodePipeline, CodeBuild, CodeDeploy: all expose detailed logs and failure reasons in CloudWatch Logs.
CloudFormation: review stack events and resource drift; CloudTrail records stack changes.

Auto Scaling: Diagnosing Failures

Auto Scaling emits scaling activity, failed launch events, and health-check metrics into CloudWatch. Use:

Scaling activity history
Lifecycle events and instance status
CloudTrail entries for API-level actions and why a scale failed

A slide titled "Troubleshooting Auto Scaling Failures" showing four colored panels — Scaling Activities, Failed Launches, Health Checks, and Scaling Policies — each with an icon and a brief tip for diagnosing auto-scaling issues.

Container Platforms: ECS & EKS

ECS and EKS surface service and task/pod events and metrics to CloudWatch (when integrated).
CloudWatch Container Insights gives node and pod-level metrics and diagnostics.
For EKS, inspect the Kubernetes control plane and use kubectl for pod-level investigation.

Useful kubectl commands:

kubectl logs <pod-name>
kubectl describe pod <pod-name>
kubectl get events

CloudWatch Container Insights maps container metrics (CPU, memory, restarts, disk IO) to containers, pods, and nodes for targeted troubleshooting.

Troubleshooting by Layer (Quick Reference)

Layer	Primary AWS Tooling	Typical Focus
Application	X-Ray, CloudWatch Logs, application logs	Request traces, error rates, business logic, memory/CPU usage
Database	RDS Performance Insights, CloudWatch, logs	Query performance, locks, resource contention
Network	VPC Flow Logs, Reachability Analyzer, CloudWatch	Packet drops, blocked routes, security group/NACL rules
AWS Service	CloudWatch metrics, AWS Health, Service Health Dashboard	Provider-side degradation, regional outages

Performance & Latency: Tools by Layer

Application: X-Ray traces, application metrics, and logs.
Database: RDS Performance Insights for slow queries and contention.
Network: VPC Flow Logs + Reachability Analyzer for dropped packets or routing issues.
AWS Service: CloudWatch metrics and AWS Health for provider-side performance issues.

A slide titled "Troubleshooting Performance and Latency Issues" showing four colorful petal-shaped icons and brief tips for the Application Layer, Database Layer, AWS Service Layer, and Network Layer (e.g., check algorithms/memory leaks, use RDS Performance Insights, review CloudWatch/AWS Health, examine VPC Flow Logs).

Network & Connectivity Checklist

When investigating connectivity failures, validate:

Security Groups: correct inbound/outbound and ports.
Network ACLs: stateless rules allow expected flows.
Route Tables: routes between subnets and gateways.
VPC Flow Logs: capture accepted vs. denied traffic for packet drop analysis.
Reachability Analyzer: simulate paths and identify blocking hops.

An infographic titled "Troubleshooting Network and Connectivity Issues" showing a numbered checklist of AWS networking checks. It lists Security Groups, Network ACLs, Route Tables, VPC Flow Logs, and Reachability Analyzer with brief guidance for each.

Permissions & IAM Troubleshooting

Common tools and techniques:

CloudTrail to see who attempted API calls and what failed.
IAM Policy Simulator to validate effective permissions for a principal.
Access Analyzer to detect overly permissive policies.
Read and capture full error messages—many API responses indicate the exact permission or resource mismatch.

A presentation slide titled "Troubleshooting IAM Permission Issues" showing four troubleshooting techniques. It lists CloudTrail, IAM Policy Simulator, Access Analyzer, and Error Messages with brief notes on each.

Anti‑Patterns to Avoid (and What to Do Instead)

Do not:

Restart systems before collecting data.
Make multiple simultaneous changes.
Ignore logs (CloudWatch) and audit trails (CloudTrail).
Skip AWS Health checks.
Operate without baseline metrics.

Instead: be data-driven, use a layered approach, document each step, and change one variable at a time.

Avoid changing multiple variables in a single step during incidents — you must be able to trace the cause and verify whether a remediation actually fixed the problem.

A presentation slide titled "Troubleshooting Anti‑Patterns to Avoid" with two columns: "Common Anti‑Patterns" and "Best Practices." The anti‑patterns list things like restarting without investigation, multiple simultaneous changes, ignoring CloudTrail, skipping AWS Health, and no baseline metrics, while the best practices list data‑driven investigation, a systematic layer approach, documenting as you go, single‑variable testing, and the 5 Whys.

When You Hit a Dead End

Ask the right diagnostic questions:

What are the exact symptoms?
What changed recently (deployments, config, IAM)?
Do metrics deviate from baseline?
Who changed configuration and when?

Follow the chain of evidence: logs, traces, metrics, and audit trails should validate or invalidate each hypothesis.

Post-Incident Review & Continuous Improvement

Make post-incident work actionable:

Run blameless postmortems focused on improvement.
Record detailed timelines, root causes, and remediation steps.
Update runbooks and playbooks with new failure modes.
Close monitoring gaps by adding alarms and metrics.
Implement preventive fixes to reduce recurrence or severity.

A slide titled "Post-Incident Review and Learning" listing review steps (focus on system improvements, record timelines/root cause, update runbooks, identify monitoring gaps, implement preventive changes). Colorful outcome boxes on the right label goals like Blameless Culture, Complete Documentation, Runbook Updates, Monitoring Gaps, and Preventive Actions.

Keeping a disciplined, data-driven approach and documenting every step reduces repeated failures and improves mean time to recovery (MTTR).

References & Further Reading

AWS CloudWatch: https://docs.aws.amazon.com/cloudwatch/
AWS X-Ray: https://docs.aws.amazon.com/xray/
AWS CloudTrail: https://docs.aws.amazon.com/cloudtrail/
AWS Health: https://status.aws.amazon.com
Systems Manager OpsCenter: https://docs.aws.amazon.com/systems-manager/latest/userguide/opscenter.html
Kubernetes kubectl reference: https://kubernetes.io/docs/reference/kubectl/overview/

That’s it for this lesson — apply the frameworks and tools above to make incident response predictable and measurable.

Introduction

Domain 1 - SDLC Automation

Domain 2 - Configuration Management and IaC

Domain 3 - Resilient Cloud Solutions

Domain 4 - Monitoring and Logging

Domain 5 - Incident and Event Response

Domain 6 - Security and Compliance

Systematic Troubleshooting

Troubleshooting Mindset: Four Core Practices

Causal Analysis: Five Whys

A Six-Step Troubleshooting Framework

Layered Analysis: Where to Look First