> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Debugging and EC2 Outage

> Guide to troubleshooting Amazon EC2 outages using status checks, CloudWatch metrics, system logs, and AWS Systems Manager, with stepwise diagnostics and corrective actions

In this demo you'll learn a practical workflow to investigate an Amazon Elastic Compute Cloud (EC2) instance outage, how to use instance diagnostics to narrow the root cause, and which corrective actions to take.

We’ll start by looking at an example of a healthy EC2 instance so you know what to expect when things are working correctly.

![A screenshot of the AWS EC2 Instances console showing one instance named "WebServerA." The instance (i-0d96cc656ab9a987c) is running as a t3.micro with 3/3 status checks passed in the us-east-2a availability zone.](https://mintcdn.com/kodekloud-c4ac6d9a/NJ9FmMaxSEmD0TY-/images/AWS-Certified-DevOps-Engineer-Professional/Domain-5-Incident-and-Event-Response/Demo-Debugging-and-EC2-Outage/aws-ec2-webservera-t3micro-use2a.jpg?fit=max\&auto=format\&n=NJ9FmMaxSEmD0TY-\&q=85\&s=8fdfdc6bc2d3b809f81106eec27d5f12)

The instance state is “running” and all 3 status checks have passed — this is the ideal state. To begin troubleshooting a failing instance, select it from the console and open the instance details panel, then expand the **Status and alarms** (Status checks) area. This panel centralizes status checks, metrics, and any associated alarms.

## Understand EC2 status checks

EC2 status checks are layered so you can quickly locate whether the issue is infrastructure, instance-level, storage, or application-specific.

| Status check | What it evaluates | Typical causes |
| -: | - | - |
| `System status` | The underlying AWS host and network that runs the instance | Host hardware failure, host networking, hypervisor issues |
| `Instance status` | The guest OS and instance-level configuration | Boot failures, kernel panics, misconfigured networking, failed services |
| `EBS status` | Reachability and I/O ability of attached EBS volumes | Detached volumes, I/O errors, corrupted filesystems |
| `Application status` | Optional custom checks that validate the app running on the instance | Application crashes, failing health checks, dependency failures |

These checks let you eliminate layers quickly: infrastructure → OS/configuration → storage → application.

## Correlate status checks with CloudWatch metrics and alarms

The instance details show related CloudWatch metrics such as:

* `StatusCheckFailed_System`
* `StatusCheckFailed_Instance`
* `StatusCheckFailed_EBSVolume`
* `StatusCheckFailed_Application` (if configured)

Use the metric time series and alarm history to determine when failures started and whether they are transient or recurring. If you already have CloudWatch alarms, they appear in the console and can trigger automated actions (for example, recovery, notifications, or Lambda workflows).

## Retrieve system logs and run in-guest diagnostics

When `Instance status` fails, start by collecting the system log and any instance-level diagnostics:

* From the console: use **Instance Settings → Get System Log** (or "Get system log") to download or copy the boot/system log.
* For Linux instances: use the serial console (if enabled) or AWS Systems Manager Session Manager (requires SSM agent and proper IAM permissions) to run diagnostics without SSH. See: [https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html)
* For Windows instances: use **Get instance screenshot** in the EC2 console or Systems Manager tools for remote diagnostics.

Example system log snippet:

```text theme={null}
[    5.259493] cloud-init[2007]: ci-info: | 1 | 172.31.0.0 | 0.0.0.0 | 255.255.240.0 | ens5 | U |
[    5.261245] cloud-init[2007]: ci-info: | 2 | 172.31.0.1 | 0.0.0.0 | 255.255.255.255 | ens5 | UH |
[    5.263053] cloud-init[2007]: ci-info: | 3 | 172.31.0.2 | 0.0.0.0 | 255.255.255.255 | ens5 | UH |
[    5.264800] cloud-init[2007]: ci-info: +-------+----------------------+----------------------+-------+----+
[    5.266508] cloud-init[2007]: ci-info: ++++++++++++++++Route IPv6 info++++++++++++++++++++++++++++++
[    5.267944] cloud-init[2007]: ci-info: +------------------------------------------------------------+
[    5.269521] cloud-init[2007]: ci-info: | Route | Destination | Gateway | Interface | Flags |
```

System logs often reveal boot errors, missing drivers, service failures, or filesystem problems that point to an `Instance status` failure.

## Investigation workflow — step by step

1. Identify which status check failed
   * `System status` → likely AWS-host or network issue.
   * `Instance status` → guest OS or instance-level configuration.
   * `EBS status` → storage or volume attachment/I/O problems.
   * `Application status` → application health checks or dependencies.

2. Correlate with CloudWatch
   * Inspect the metric time series for the failing check.
   * Review alarm history and triggered actions to understand timing and recurrence.

3. Collect diagnostics
   * Download the system log.
   * Connect using Session Manager or the serial console for interactive troubleshooting.
   * For Windows, collect screenshots and event logs via Systems Manager.

4. Apply corrective actions based on the failure type
   * System host/network (`System status`):
     * Consider enabling EC2 Auto Recovery via a CloudWatch alarm on `StatusCheckFailed_System`.
     * Validate instance type and region support for auto recovery before relying on it.
   * Guest OS (`Instance status`):
     * Inspect logs, repair boot configuration, revert recent changes, or detach the root volume and mount it on another instance for offline repair.
   * Storage (`EBS status`):
     * Check volume attachment state, run filesystem checks (`fsck` for Linux), review snapshots, and restore or replace volumes if needed.
   * Application (`Application status`):
     * Review application logs, health endpoints, dependencies (DB, cache), and configuration; fix at application level.

## Recommended corrective-action matrix

| Failure type | Quick checks | Typical corrective actions |
| - | - | - |
| `System status` | Console host events, CloudWatch `StatusCheckFailed_System` | Auto recovery, contact AWS Support, replace instance |
| `Instance status` | System log, serial console, SSM session | Repair boot, undo config changes, offline root-volume repair |
| `EBS status` | Volume state, I/O errors, snapshots | Reattach/replace volume, restore from snapshot, run `fsck` |
| `Application status` | App logs, health checks, dependencies | Fix app config, restart service, scale/rollback deployment |

When possible, instrument instances with [AWS CloudWatch](https://learn.kodekloud.com/user/courses/aws-cloudwatch) and SSM (Session Manager) ahead of time. These tools greatly simplify post-failure investigation and enable non-disruptive diagnostics (for example, retrieving logs or running commands without SSH). See AWS Session Manager docs: [https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html)

## Important caveat about automatic recovery

Auto recovery actions (for example, CloudWatch alarm -> "Recover this instance") are useful for many host-level failures, but they have preconditions: not all instance types or regions support automatic recovery, and some recovery actions may change the underlying host (new host permissions, new hardware). Validate compatibility and have backup strategies (AMIs, snapshots) before relying on automation.

## Summary checklist

* Start at the **Status and alarms** area to identify the failing layer.
* Use CloudWatch metrics and alarm history to understand timing and frequency of failures.
* Retrieve system logs and use SSM/serial console for in-guest diagnostics.
* Apply corrective actions mapped to the failing layer: host recovery, in-guest fixes, storage restore, or application fixes.
* Instrument instances proactively with CloudWatch and SSM to speed future investigations.

## Links and references

* AWS Systems Manager Session Manager: [https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html)
* EC2 status checks and troubleshooting (AWS Docs): [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html)
* AWS CloudWatch alarms: [https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html)

That's the recommended troubleshooting flow for diagnosing and responding to failed EC2 instances and related outages. Use the layered approach to isolate the root cause quickly and apply the appropriate corrective action.

> **[Watch Video](https://learn.kodekloud.com/user/courses/aws-certified-devops-engineer-professional/module/0054c8f3-16fd-4414-be5f-d43fc173b566/lesson/719a3217-5dae-425f-8893-a19e771d24de)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/aws-certified-devops-engineer-professional/module/0054c8f3-16fd-4414-be5f-d43fc173b566/lesson/f8862920-0f60-44c2-8862-a0e5b5435f8f)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.