Skip to main content
This lesson reviews core AWS services and architectural choices for deploying machine learning models at scale. Amazon SageMaker is central to many options, surrounded by endpoint types, model packaging and registry, CI/CD and orchestration, monitoring, and autoscaling strategies. Below we cover four primary SageMaker deployment patterns, how to package and register models, infrastructure choices, deployment strategies, CI/CD and orchestration options, monitoring best practices, and a concise decision guide.

​
SageMaker deployment patterns

Choose the deployment pattern that matches your SLA, payload, throughput and cost constraints:
  • Real-time endpoints: low-latency, always-on serving using CPU/GPU instances (often in a VPC) with autoscaling for steady traffic.
  • Asynchronous inference: for large payloads or long-running inference tasks where responses are returned later.
  • Batch transform: high-throughput, offline processing where latency is not critical.
  • Serverless inference: handles spiky workloads without provisioning instances; good for unpredictable traffic and lower management overhead.
The image outlines deployment patterns for machine learning, including Batch Transform, Async Inference, and Real-Time Endpoints, each with brief descriptions of their characteristics.

​
Model packaging and registry flow

Standard packaging and registry steps ensure reproducibility and traceability:
  1. Upload the trained model artifact (for example, model.tar.gz) to Amazon S3.
  2. Build the container image (inference runtime) and push it to Amazon ECR.
  3. Create a SageMaker Model object that references the S3 model artifact and the ECR image.
  4. Register the Model in the SageMaker Model Registry for versioning, approval workflows, and lifecycle management.
Use the Model Registry to gate deployments via approvals, metadata, and automated validation steps.
The image is a diagram illustrating the flow of model packaging and registry, involving S3, ECR, and Amazon SageMaker Model.

​
Endpoint types and infrastructure choices

Match endpoint type to latency, payload size, and scaling requirements. Links for deeper reading:
The image is a table summarizing endpoint types and infrastructure choices, with columns for endpoint type, infrastructure choice, and when to use each type.

​
Deployment strategies

Traffic is typically routed via an API Gateway or an Application Load Balancer to the active environment (blue). New model versions are deployed as candidates (green). Common strategies:
  • Blue/Green: run a fully provisioned stable (blue) environment while validating a candidate (green) environment before switching traffic.
  • Canary: gradually shift a small percentage of traffic to the new version (e.g., 5% → 25% → 100%), monitor metrics, then promote or rollback.
  • Shadow: mirror production traffic to the new model without returning its responses to users—useful for validating behavior on real traffic.
  • Rollback: define automated or manual rollback paths if metrics or alarms indicate regressions.
Consider instrumenting canary rollouts with automated metric comparisons and alerting for faster rollback decisions.
The image is a diagram illustrating deployment strategies such as Blue/Green, Canary, Shadow, and Rollback. It shows a flow from a client to an API Gateway/ALB, leading to different environments: Blue (current), Green (candidate), and Shadow.
Canary rollouts let you promote a new model safely by incrementally increasing traffic while continuously monitoring metrics to detect regressions and trigger rollbacks when necessary.

​
CI/CD and MLOps

A robust MLOps pipeline ties source control, testing, orchestration, and deployment together:
  • Source: push code and model artifacts to a Git provider (CodeCommit, GitHub, etc.).
  • Build & test: run unit tests, integration tests, and data checks using CodeBuild or other CI runners.
  • Orchestration: use CodePipeline to organize CI/CD stages; use SageMaker Pipelines for ML-specific workflows (training, evaluation, registration, deployment).
  • Model Registry: after passing validations, register models in the SageMaker Model Registry to control approvals and deployments.
Event-driven orchestration patterns:
  • EventBridge: trigger workflows on events such as “model registered”, job failures, or scheduled tasks.
  • Step Functions: orchestrate complex stateful workflows — e.g., preprocessing → training → evaluation → registration → deployment.
  • SageMaker Pipelines / Jobs: execute training, processing, and deployment tasks as discrete, auditable steps.
The image illustrates an orchestration and automation process involving EventBridge for triggers, Step Functions as a state machine, and SageMaker Pipelines or Jobs.

​
Monitoring, alerts, and anti-patterns

Collect logs and metrics from endpoints, batch jobs, and containers, then surface anomalies with alarms:
  • Use CloudWatch logs/metrics and container logs for observability.
  • Use SageMaker Model Monitor to detect data drift, changes in feature distributions, and prediction-quality issues.
  • Wire CloudWatch alarms and EventBridge to trigger automated responses (retraining, rollback, paging on-call).
Common anti-patterns to avoid:
  • Monolithic endpoints that prevent independent updates and increase the blast radius.
  • Overly permissive IAM policies that create security and compliance risks.
  • Using long-lived, provisioned real-time endpoints for highly spiky traffic when serverless or async inference would be more cost-effective.
Do not rely solely on periodic manual checks. Lack of automated monitoring and alerts creates blind spots where regressions or data quality issues can go undetected. Ensure Model Monitor and CloudWatch alarms are part of your deployment gates.
The image is a flow diagram illustrating a monitoring system involving AWS services like SageMaker Model Monitor, CloudWatch Alerts, and EventBridge. It shows the process from endpoint or batch data to logs and metrics, through model monitoring for data drift checks, then to alerting and event management.

​
Decision guide (summary)

Start by defining SLA and non-functional requirements: latency, throughput, payload size, cost budget, and security constraints.
  • Low latency & steady traffic: SageMaker real-time endpoints or a managed Amazon EKS serving solution.
  • Offline, high-throughput jobs: Batch Transform or AWS Batch.
  • Large payloads or long-running inference: SageMaker Asynchronous Inference.
  • Unpredictable, spiky traffic with minimal infra ops: choose serverless (SageMaker Serverless Inference or AWS Lambda for simple functions).
Consider automating validation and monitoring in your CI/CD pipeline: register models only after automated evaluation, use canary or shadow deployments for validation, and wire Model Monitor and alarms into your deployment decision gates.

Watch Video