KodeKloud Notes

When your application performs an action—such as a user login or a data update—it generates a log event, capturing metadata and a message about that action. AWS CloudWatch Logs ingests millions of log events per second by organizing them into log streams, which are then grouped under log groups.

The image is a diagram explaining the flow of log events in Amazon CloudWatch Logs, showing how log events are organized into log streams and grouped under a log group.

A log group is a container for one or more log streams. Each log stream represents an ordered sequence of log events from the same source—for example, an EC2 instance, Lambda function, or on-premises server.

Anatomy of a Log Event

Each log event in CloudWatch Logs is usually a JSON object with the following core attributes:

{
  "id": "12345678-1234-1234-1234-123456789012",
  "timestamp": 1677647621000,
  "message": "User login successful: username=johndoe",
  "logGroupName": "/aws/ec2/my-application",
  "logStreamName": "2023/10/20/instance-i-0abcd1234efgh5678",
  "source": "my-application",
  "instanceId": "i-0abcd1234efgh5678",
  "eventSource": "application",
  "eventType": "UserLogin",
  "applicationVersion": "1.0.0",
  "region": "us-east-1"
}

Here, eventType is UserLogin, indicating a successful login. Other common event types might include PasswordReset, LoginFailure, or AccountLockout.

Structured vs. Unstructured Log Events

CloudWatch Logs supports both structured and unstructured log data:

Log Type	Format	Benefits	Use Cases
Structured	JSON, key–value	Easy search & parse, consistent schema	API request logs, audit trails, metrics
Unstructured	Plain text	Flexible, minimal setup	Error stack traces, debug output

Note

Structured logs simplify querying with CloudWatch Logs Insights and help maintain consistent event schemas.

Immutability of Log Events

Once ingested, log events in CloudWatch Logs are immutable—you cannot alter or delete individual records. To remove data, you must delete the entire log stream or log group.

The image explains log events, showing a log stream with icons and a section on immutable log events, highlighting their immutability, inability to be deleted or modified, and ensuring data integrity.

Warning

Immutable logs ensure data integrity and compliance. Plan your retention policies carefully: removing sensitive data requires deleting the whole stream or group.

Pre-Ingestion Filtering

To reduce costs and enhance signal-to-noise in your logs, apply pre-ingestion filtering. This lets you filter, transform, or drop unwanted log data on the client side before sending to CloudWatch Logs.

The image illustrates the concept of understanding log events, showing a log stream with icons representing logs and a section on pre-ingestion filtering, which includes filtering and processing log data on the client side before sending to CloudWatch Logs.

Key benefits of pre-ingestion filtering:

Lower storage and ingestion costs
Focus on critical events
Improved performance when querying

Note

Use the AWS SDK or CloudWatch Logs Agent to configure filters before data is sent.

Summary

By organizing log events into streams and groups, choosing structured or unstructured formats, leveraging immutability, and filtering at the source, you can build an efficient, secure, and cost-effective logging strategy with AWS CloudWatch Logs.

Links and References

Watch Video

Watch video content