Shared responsibility model: who owns what
Understanding the shared responsibility model is the first step to framing resiliency responsibilities between you and AWS. The level of operational responsibility shifts depending on the service model:
The higher the level of managed service, the less you manage low-level reliability — but you still design for availability, change management, and failure response.
Four core focus areas for reliable systems
These four focus areas form the foundation of reliability and should guide architecture and operational decisions:- Infrastructure architecture (the foundation)
- Application and service architecture
- Change management
- Failure management
1. Infrastructure architecture (the foundation)
Key goals: eliminate single points of failure, ensure capacity, and plan resilient connectivity.- Track and manage service quotas (limits) in the AWS Console. Quotas protect against accidental overconsumption (for example, spinning up an unexpected number of instances). Request increases proactively before hitting production demand.
- Account for provider and third-party constraints (network throughput, file descriptors, database connections).
- Architect redundant network paths and multi-AZ or multi-region endpoints where required for availability.
- Choose an IP addressing and topology pattern that fits your availability and security goals (hub-and-spoke, transit gateway, or full mesh).
- Treat storage durability explicitly: use appropriate replication and snapshots. Example services:
- Amazon EBS: durable block storage with availability guarantees; combine with snapshots for backups.
- Amazon Aurora: multi-copy replication across nodes and AZs for higher availability.
2. Application and service architecture
Design applications so failures are contained, visible, and recoverable.- Segment systems by domain or bounded context (e.g., events, profile, orders) to reduce blast radius.
- Prefer loose coupling (message queues, event-driven patterns) so services fail independently.
- Support graceful shutdown and stateless design where possible; for stateful components, use replication and automatic failover.
- Emit telemetry: metrics, distributed traces, and structured logs from each component for fast detection and diagnosis.
- Keep changes small and focused to reduce rollout risk and make rollbacks simpler.
3. Change management
Reduce human error and deployment risk via automation and safe rollout patterns.- Automate deployments, tests, and rollbacks using CI/CD pipelines.
- Use canary or blue/green deployments to limit impact of a failed change and to validate behavior under production traffic.
- Maintain runbooks and rehearse deployment procedures; validate rollback steps frequently.
- Automate post-deployment verification (smoke tests, health checks, synthetic transactions).
4. Failure management (prepare and rehearse)
Prepare for failure rather than hoping it never occurs.- Implement regular backups and automate restore verification. Test full restores to validate integrity and recovery time objectives.
- Perform capacity and performance tests under realistic load to validate autoscaling and throttling behavior.
- Use chaos engineering and game days to intentionally inject faults and validate recovery procedures. Practice taking components offline and observe system behavior.
- Ensure Auto Scaling and instance replacement strategies are configured for compute failures; design application-level failover for managed-service outages.
Quotas, capacity, and graceful degradation
Numerical constraints (quotas and throughput limits) directly shape architecture and operational choices.- Monitor key limits: concurrent connections, network throughput, request rate, and quota counts.
- Request increases in advance of expected demand. Design for graceful degradation when limits are reached (rate limiting, backpressure, fallbacks).
- Include limit thresholds in your monitoring and alerting to avoid sudden service disruption.
Testing for reliability
Testing is the only way to verify that reliability designs actually work.- Include functional, load, and resilience tests in your CI/CD pipeline.
- Automate teardown and resource cleanup so tests do not leave stale, costly resources.
- Test real-world failure scenarios: AZ failures, network partitions, instance replacement, and degraded dependencies.
Quick reference: Reliability patterns
Summary
- The shared responsibility model sets clear boundaries: AWS manages infrastructure and many platform responsibilities; you design and operate applications, data, and integrations.
- Focus on four foundations — infrastructure, application architecture, change management, and failure management — to build resilient systems on AWS.
- Design for failure, test frequently, automate rollouts and rollbacks, and rehearse recovery procedures.
Remember: design for failure and test frequently. Automated testing, monitoring, and small, safe rollouts significantly reduce the time and impact of real-world incidents.