Demo Cloudwatch agent to setup SSH connection failure alert dashboard for EC2 instance

In this guide, you’ll learn how to install and configure the AWS CloudWatch Agent on an EC2 instance to collect SSH login and audit logs, stream them to CloudWatch Logs, and verify them in the console. Once streaming is in place, you can create metric filters, alarms, and dashboards to monitor SSH connection failures and other security events.

Table of Contents

1. Prerequisites
2. Update IAM Role
3. Launch an EC2 Instance
4. Verify Instance Status
5. Inspect Audit Logs on EC2
6. Install the CloudWatch Agent
7. Configure Log Collection
8. Create the CloudWatch Log Group
9. Start and Validate the CloudWatch Agent
10. View Logs in CloudWatch
11. Next Steps
12. References

Prerequisites

• An existing IAM role (e.g., metrics-filter) with console access.
• An AWS account with permissions to manage EC2, IAM, and CloudWatch.
• A security group that allows SSH (port 22).

1. Update IAM Role

Attach the CloudWatchAgentServerPolicy to your IAM role to grant permission for log streaming and metrics.

Steps:

1. Open the IAM console and choose Roles.
2. Select metrics-filter, then Add permissions → Attach policies.
3. Search for and attach CloudWatchAgentServerPolicy.

2. Launch an EC2 Instance

1. In the EC2 console, click Instances → Launch instances.
2. Select an Amazon Linux AMI and an appropriate instance type.

3. Warning

   For demonstration only: you may proceed without a key pair. Do not skip key pair selection in production.

4. Under Network settings, choose your security group (allow SSH).
5. Expand Advanced details and assign the updated IAM role (metrics-filter).
6. Click Launch instance.

3. Verify Instance Status

Wait for your instance to enter the running state and pass status checks.

4. Inspect Audit Logs on EC2

SSH into the instance, switch to root, and explore the audit logs:

ssh ec2-user@<instance-ip>
sudo su -
cd /var/log
ls -l
tail -100f audit/audit.log

You should see entries for SSH logins, sudo commands, and other audit events.

5. Install the CloudWatch Agent

Download and install the agent package:

cd ~
wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
unzip AmazonCloudWatchAgent.zip
sudo ./install.sh

This process creates the cwagent user and group.

6. Configure Log Collection

Create cloudwatch-agent-config.json in your home directory:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/audit/audit.log",
            "log_group_name": "login-monitoring",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

7. Create the CloudWatch Log Group

1. Open the CloudWatch console and go to Logs → Log groups.
2. Click Create log group, name it login-monitoring, and configure retention as needed.

No manual log streams needed: the agent creates one per EC2 instance.

8. Start and Validate the CloudWatch Agent

Fetch the configuration and launch the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 \
  -cf file:cloudwatch-agent-config.json -s

Check status:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

Expected output:

{
  "status": "running",
  "starttime": "2023-11-30T02:41:10+00:00",
  "configstatus": "configured",
  "version": "1.30001.0b313"
}

Inspect agent logs:

ls -l /var/log/amazon-cloudwatch-agent
ls -l /opt/aws/amazon-cloudwatch-agent/logs
tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

9. View Logs in CloudWatch

Back in the CloudWatch console, navigate to Logs → Log groups → login-monitoring and refresh. You’ll see one log stream per instance.

Click your instance’s log stream to inspect log events:

Next Steps

• Create metric filters to detect failed SSH attempts:

  aws logs put-metric-filter \
    --log-group-name login-monitoring \
    --filter-name SSHFailFilter \
    --filter-pattern "{ $.message = *Failed password* }" \
    --metric-transformations \
      metricName=SSHFailCount,metricNamespace=Security,metricValue=1
  

• Set up CloudWatch Alarms on SSHFailCount.
• Build a dashboard to visualize login attempts and failures.

References

• AWS CloudWatch Logs Documentation
• Amazon EC2 User Guide
• AWS CLI Command Reference