Skip to main content
Welcome. This lesson explains how to design for reliability for data, machine learning, and AI services on AWS. It highlights the reliability controls you can configure, what AWS handles for you, and the architectural patterns to apply around managed APIs to build resilient pipelines.
This guide focuses on operational levers (scaling, replication, retention, monitoring) and orchestration patterns (buffering, retries, multi‑AZ/multi‑region) you can use to improve availability, durability, and recoverability for AWS data and ML/AI services.

​
Kinesis Data Streams — data-in-transit (shards, autoscaling, retention)

Kinesis Data Streams is AWS’s managed streaming service for data-in-transit. Its unit of scale is the shard: each shard provides a fixed amount of read and write throughput and you scale a stream by adding or removing shards. When a shard splits, resulting child shards manage throughput independently.
A diagram illustrating AWS Kinesis shard lifecycle: an open shard is closed as a parent shard and splits into two open child shards containing records. The caption notes Kinesis handles data-in-transit with built-in redundancy and automatic scaling.
Key operational knobs:
  • Shard autoscaling (based on utilization) to adapt to workload spikes and drops.
  • Capacity mode selection at stream creation: Provisioned or On‑demand.
  • Retention period: configurable from 24 hours up to 365 days.
Typical shard throughput:
  • Write: up to 1 MB/sec or 1,000 records/sec per shard.
  • Read: up to 2 MB/sec per shard.
When to use which capacity mode:
  • Provisioned: use when your traffic is predictable and you want precise throughput control.
  • On‑demand: use when workloads are unpredictable or have bursty spikes; AWS manages scaling.
Manage quotas and retention:
  • Account and regional quotas can limit shard counts and aggregate throughput — request increases via Service Quotas when needed.
  • Increase retention if downstream processing delays exceed the default retention window to avoid data loss.
A presentation slide titled "Designing for Reliability – Kinesis" showing a screenshot of the AWS Kinesis "Create data stream" configuration page. The screenshot highlights capacity options for On-demand and Provisioned modes.
Example CLI for extending retention:
A slide titled "Designing for Reliability – Kinesis" showing the AWS Kinesis console screen for editing a stream's data retention period, with radio options from 1 day up to 365 days (7 days selected) and Cancel/Save changes buttons.
Beware of exceeding shard quotas or retention costs. Long retention increases storage costs; many teams use on‑demand shards combined with durable S3 backup or DynamoDB checks to balance cost and recovery needs.

​
Managed Streaming for Kafka (MSK)

Amazon MSK provides managed Apache Kafka with broker nodes, Zookeeper (for older Kafka versions), and network entry points. Producers write to brokers; data is replicated and consumed by clients.
A technical architecture diagram titled "Designing for Reliability – Managed Service for Kafka" showing Amazon MSK deployed across three availability zones with Zookeeper nodes, Kafka brokers, ENIs, private subnets, and connections to producers, consumers, and cluster operations. The caption at the bottom reads "Zookeeper integrates natively."
Reliability considerations:
  • Deploy brokers across multiple Availability Zones to achieve fault tolerance.
  • Configure replication factor and in-sync replicas (ISRs) to balance durability and write latency.
  • MSK supports automatic broker replacement and can automate certificate renewals and maintenance.
  • For cross-region replication, use MirrorMaker or other async replication strategies.
  • Newer Kafka KRaft modes reduce or remove ZooKeeper dependency; MSK supports evolving architectures as available.
Recommended actions:
  • Set appropriate acks on producers (e.g., all for durability).
  • Monitor broker health, topic lag, and ISR counts with CloudWatch metrics.
  • Plan maintenance windows and use MSK’s automated patching where acceptable.

​
AWS Glue — serverless ETL (Apache Spark)

AWS Glue is a serverless ETL service built on Apache Spark that crawls data, builds the Glue Data Catalog, and runs ETL jobs to transform and move data (commonly into S3 for Athena/QuickSight).
A slide titled "Designing for Reliability – Glue" showing an AWS Glue data pipeline diagram with icons for Amazon RDS (Aurora-MySQL), AWS Glue crawlers, the Glue Data Catalog, Glue ETL, an S3 bucket, and Amazon Athena connected by numbered arrows. It illustrates the flow of data and processes between these AWS services.
Glue reliability knobs:
  • Retries: enable automatic retries for transient job failures.
  • DPUs (Data Processing Units): scale compute per job; right-size DPUs to match job complexity and cost goals.
  • Use job bookmarks/state to avoid reprocessing and reduce failure blast radius.
Best practices:
  • Tune DPUs and monitor job run times and errors with CloudWatch.
  • Use incremental ETL patterns and idempotent transforms to support safe retries.

​
Amazon EMR — big data processing clusters

EMR is a node-based, managed platform for Hadoop, Spark, HBase, and more. Reliability controls:
  • Configure multiple master nodes or automatic failover for NameNode redundancy.
  • Tune YARN for resource allocation and reliable Spark execution.
  • Spread clusters across Availability Zones when feasible.
  • Use EMRFS to interact with S3 for durable storage and checkpointing.
Operational tips:
  • Integrate EMR with CloudWatch, CloudTrail, and AWS Config for monitoring and auditing.
  • Enable SSH access to inspect logs on core/task nodes when troubleshooting.

​
Glue DataBrew — visual data preparation

Glue DataBrew is a fully managed visual data-prep service with few operational knobs — AWS handles scaling and infrastructure. Focus on job-level settings and retries.
An AWS architecture diagram titled "Designing for Reliability – Glue DataBrew" showing a data pipeline where solar panel datasets land in Amazon S3 (/raw-data → /transformed-data → /curated-data), are processed and cataloged with AWS Glue and Glue DataBrew, and then queried by Amazon Athena and visualized in Amazon QuickSight for business analysts.

​
Lake Formation — unified data lake management

Lake Formation orchestrates S3, the Glue Data Catalog, and IAM to build secure data lakes. It relies on the reliability of underlying services like S3, Glue, and EMR.
A slide titled "Designing for Reliability – Lake Formation" showing an AWS data lake architecture with Amazon Lake Formation, S3 drop/analytics/curated zones, AWS Glue, EMR, Lambda, Step Functions, Athena, QuickSight and API Gateway connected by data flow arrows. The slide also includes a note saying Lake Formation is highly reliable but provides few levers for reliability itself.
Design guidance:
  • Rely on S3 durability for storage; enforce fine‑grained access via Lake Formation and IAM policies.
  • Use catalog versioning and replication strategies for cross-region resilience if required.

​
Athena — serverless interactive queries

Athena is a serverless SQL query service (based on Presto) that queries data in S3 and federated sources.
A presentation slide titled "Designing for Reliability – Athena" showing Amazon Athena as a serverless analytics service in the center, ingesting data from sources like operational databases, data warehouses, big data, ERP and Amazon S3, and feeding ML tools and BI/analytics applications. The left caption notes Athena is a high‑level, serverless, highly available service backed by S3.
Focus areas for reliability:
  • Ensure S3 data durability and correct IAM permissions for query access.
  • Partition and compress data to improve query performance and reduce failures due to timeouts.
  • Monitor query metrics and logs in CloudWatch and use cost controls (query limits) to prevent noisy neighbors.

​
QuickSight — managed BI and dashboards

QuickSight is a managed BI tool with autoscaling and a SPICE in-memory engine.
A slide titled "Designing for Reliability – Quicksight" showing a simple architecture diagram with a client laptop, web server, and a Quicksight engine. The diagram highlights autoscaling and connections from Quicksight to SPICE storage and a direct-query database.
Operational focus:
  • Manage SPICE refresh schedules and dataset size.
  • Secure access and permissions; monitor dashboard health and query times.

​
Redshift — data warehouse

Amazon Redshift provides clustering, replication, and snapshotting for analytical workloads.
A diagram titled "Designing for Reliability – Redshift" showing multiple source systems feeding an Amazon Redshift producer that replicates raw and aggregated data. The data is shared to Redshift consumer clusters and routed via Amazon Route 53 to internal and external users.
Reliability best practices:
  • Use snapshot retention and automated snapshots for backups.
  • Consider RA3 nodes with managed storage for scaling and durability.
  • Implement cluster resizing and concurrency scaling to handle variable query loads.
  • Use cross-region snapshots for DR scenarios.

​
SageMaker — model training and hosting (instance-based)

SageMaker includes both managed and instance-based features. When using instance-based training or hosting, treat instances like EC2: you own configuration, scaling, and HA.
A diagram titled "Designing for Reliability – Sagemaker" showing Amazon SageMaker distributing training jobs across multiple P3 training instances inside a dashed boundary, with data stored in an Amazon S3 bucket. The caption notes this distribution increases reliability and speed.
SageMaker reliability recommendations:
  • Use distributed training across multiple instances for large models/datasets.
  • For hosting: place endpoints behind Application Load Balancers, use multi-AZ endpoints or multiple endpoints with route control, implement canary or blue/green deployments, and enable autoscaling.
  • Monitor training and hosting with CloudWatch, and audit via CloudTrail.
Secure/no-internet deployments:
  • For sensitive workloads, run SageMaker in a VPC without an internet gateway and use Direct Connect, VPC endpoints, and strict IAM policies.
An infographic asking how a biotech company can configure Amazon SageMaker to run in a "No Internet" mode for sensitive genomic data, with four numbered recommendations. The suggestions include using a VPC without an internet gateway, setting notebooks to offline mode, using AWS Direct Connect, and enforcing an IAM policy to block internet access.

​
Rekognition — image and video analysis

Rekognition is a managed image/video analysis API. AWS manages scaling and fault tolerance; architect around the service for resilience.
A system architecture diagram titled "Designing for Reliability – Rekognition" showing 3xLOGIC cameras sending video via an API into Amazon Kinesis and Rekognition. Outputs flow to Amazon S3 and Amazon SNS and connect to 3xLOGIC's managed video monitoring and monitoring agents.
Design patterns:
  • Durable ingestion (S3) + buffering (Kinesis or SQS) to decouple upstream devices from Rekognition availability.
  • Implement retry/backoff and dead-letter handling for failed requests.
  • Monitor API errors and latency in CloudWatch.

​
Polly, Lex, Comprehend — managed AI services

Polly (TTS), Lex (conversational bots), and Comprehend (NLP) are highly managed. Focus on instrumentation, monitoring, and resilient integration patterns. Polly architecture example:
A slide titled "Designing for Reliability – Polly" showing an AWS architecture diagram where users go through CloudFront and API Gateway to AWS Lambda functions that call Amazon Polly, with Amazon S3 hosting the React frontend. The slide text notes that Polly has metrics to track outputs but is too highly managed and offers little to increase reliability.
Lex monitoring options:
A slide asking how to reliably monitor an Amazon Lex chatbot. It lists four options: enable CloudWatch metrics, review Lex logs with CloudWatch Logs Insights, use AWS X-Ray tracing, or ping the Lex runtime API from a scheduled Lambda.
Recommendations:
  • Enable CloudWatch metrics and logging for each service.
  • Use X‑Ray for end‑to‑end tracing across Lambda/API Gateway/Lex if you need request-level diagnostics.
  • Consider synthetic checks for critical conversational flows.

​
Forecast, A2I (Augmented AI), Fraud Detector

  • Amazon Forecast: managed time-series forecasting; monitor model accuracy metrics, prediction latency, and retraining cadence via CloudWatch.
  • Augmented AI (A2I): provides human-review workflows. Reliability concerns focus on orchestration and human process availability (worker queues, scaling, notifications).
A diagram of an AWS augmented AI translation workflow. It shows input files in S3 processed by Lambda and Amazon Translate, reviewed via Amazon Augmented AI (human post-edits), then reconstructed and stored back in S3 with CloudWatch monitoring.
  • Fraud Detector: a managed anomaly/fraud service — architect ingestion, queueing, and human-review around it for end-to-end resilience.
A system architecture diagram titled "Designing for Reliability – Fraud Detector" showing a client app sending transaction data to a fraud detector model that returns low-risk transactions to the client and routes high-risk ones for human review. Review results are stored in Amazon S3 and fed back as training data for continuous model improvement.

​
Transcribe and Translate

Amazon Transcribe and Translate are fully managed APIs. For reliability, focus on:
  • Request retry logic and exponential backoff in clients.
  • Buffering and queuing (SQS/Kinesis) for bursts.
  • CloudWatch metrics and alarms for latency and error rates.

​
Textract — document data extraction

Textract extracts text, forms, and tables from documents. Typical pipelines use S3 for ingestion, Textract for extraction, a rules engine (plus optional A2I human review), then downstream storage/visualization.
A flow diagram of an Amazon Textract pipeline showing documents from an S3 bucket being processed to extract text, validated by a rules engine (with optional human review via Amazon A2I), and then forwarded to a dashboard for visualization. The image labels the three stages as Extraction, Validation, and Visualization.
Operational guidance:
  • Use S3 notifications + Lambda workflows for asynchronous processing.
  • Add retries with dead-letter queues for failed extractions.
  • Enable CloudWatch metrics and optionally X‑Ray for tracing complex pipelines.

​
Summary — where the reliability levers are

Use the following table to quickly identify which services expose reliability controls you manage versus those that are highly managed by AWS. Operational practices that apply everywhere:
  • Use CloudWatch for metrics, alarms, and logs.
  • Use CloudTrail for auditing and investigating API changes.
  • Implement retries with exponential backoff and idempotency to avoid duplicate processing.
  • Design multi‑AZ or multi‑region failover when service and workload requirements justify it.
  • Right‑size compute (DPUs, instance types, shard counts) to balance performance and cost.
Designing for reliability in data and ML/AI services is about understanding the service boundary — what AWS guarantees and what you must engineer — then applying buffering, retries, monitoring, replication, and right‑sizing to build robust pipelines.

Watch Video