Turning up Reliability on Compute Services Part 1

​

Designing resilient EC2 deployments

• Run multiple instances (Auto Scaling across AZs) to avoid a single instance failure.
• Use Auto Recovery for impaired instances when supported (EBS-backed in a VPC) to keep the same instance ID and private IP.
• Prefer immutable infrastructure: build and bake AMIs, replace instances instead of patching them in-place.
• Use Fast Snapshot Restore for faster EBS boot and improved I/O performance from restored snapshots in specific Availability Zones.

• Auto Scaling groups (ASG) replace failed instances automatically. Even a single-instance ASG can keep a service available by launching a replacement.
• EC2 Auto Recovery attempts to recover impaired instances; when recovery works, instance IDs and private IPs remain unchanged.
• Enable EC2 Fast Snapshot Restore (FSR) for boot snapshots in AZs where you will restore frequently — this reduces boot and I/O latency compared with standard snapshot restores.
• Immutable infrastructure and baked AMIs: create tested AMIs and launch new instances rather than performing complex boot-time configuration.

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name my-asg \
  --launch-template LaunchTemplateId=lt-0123456789abcdef0,Version=1 \
  --min-size 1 --max-size 3 --desired-capacity 1 \
  --vpc-zone-identifier "subnet-12345678,subnet-23456789"

​

Loose coupling and queues

• Synchronous decoupling: load balancers and timeouts — a client waits for a response.
• Asynchronous decoupling: queues (SQS) and topics (SNS) — allow fire-and-forget or deferred processing.

• Upload → S3 generates a notification (SNS/S3 event) and sends a message to SQS.
• EC2 workers pull messages from SQS to process videos.
• Resiliency considerations:
  • Make processing idempotent — record message IDs or processing markers in a database so retries don’t produce duplicate results.
  • Use SQS visibility timeouts and dead-letter queues (DLQs) to avoid stuck or poison messages.
  • Autoscale worker fleet based on queue depth with CloudWatch metrics.
  • Use SNS to fan-out notifications to multiple SQS queues to support parallel processing or different consumers.

​

Image Builder, Launch Templates, and Auto Scaling refresh

• EC2 Image Builder automates AMI creation and testing as part of a repeatable pipeline.
• Integrate Image Builder with SNS/Lambda to automate launch template versioning and trigger Auto Scaling instance refreshes.
• Use Launch Templates with parameters and overrides to ensure consistent instance configurations across environments.

​

Consistent EC2 deployments across environments

• Use EC2 Launch Templates and parameterize differences (instance type, user data overrides).
• Automate AMI copying across accounts and regions as part of your release pipeline.
• Prefer baked AMIs for fast, predictable boot behavior; use instance user data for light bootstrap tasks only.

​

Monitoring and metrics

• Install and configure the CloudWatch agent on EC2 instances to collect OS-level and custom metrics.
• Stream logs to CloudWatch Logs; use CloudWatch Embedded Metric Format (EMF) to extract metrics from logs.
• Push application-specific metrics to CloudWatch via the PutMetricData API when necessary.
• Use the AWS CLI/SDK to pull metrics or integrate metric export into dashboards.

• Target tracking policies (for example, maintain average CPUUtilization near 40%) provide stable automated scaling behavior.
• Scheduled and predictive scaling are useful when you have consistent patterns and historical data (predictive scaling requires adequate historic metrics to forecast demand).
• CloudWatch alarms can trigger scaling policies for reactive scenarios.

• Use a target-tracking Auto Scaling policy bound to CPUUtilization or a custom metric to keep average utilization near the desired percentage.
• Complement target tracking with scheduled scaling for predictable load spikes and predictive scaling when historical metrics provide reliable forecasts.

​

Change management and auditing

​

Instance types and reliability

• For most instance families, instance type does not materially affect basic reliability — requested vCPU and RAM are delivered as specified.
• Exception: T-family burstable instances use CPU credits and can throttle sustained CPU workloads — not suitable for consistent, CPU-bound services.
• For tenancy/isolation requirements, consider Dedicated Hosts (physical host allocation with placement visibility) or Dedicated Instances (instances on dedicated hardware).

​

Placement groups and instance placement

​

EC2 Image Builder (again — pipelines and gold images)

• EC2 Image Builder is a managed service for producing standardized AMIs (golden images).
• It improves consistency and repeatability in your deployment pipeline; treat it as part of your CI/CD infrastructure rather than a runtime reliability control.

​

Elastic Beanstalk

• Blue/green and immutable deployments minimize downtime by creating new capacity before switching traffic.
• Rolling and all-at-once deployments risk partial or full downtime during updates (all-at-once is highest risk).
• Choose deployment policies based on availability requirements and risk tolerance.

• Integrate application instrumentation (for example AWS X-Ray) to visualize service maps, latencies, and error traces across the stack and accelerate root-cause analysis.

​

Amazon Lightsail (honorable mention)

• Lightsail is a simple VPS-style service useful for small workloads and prototyping. It provides fewer low-level controls than EC2 and is typically out of scope for Solutions Architect Associate reliability questions.
• Use Lightsail for simplicity, EC2 for production workloads requiring fine-grained reliability and operational control.

​

Next: containers