Skip to main content
This lesson explains choices and trade-offs for moving machine learning (ML) models from development into production on AWS. It covers exam-relevant services and tasks including endpoints (real-time, batch, async, serverless), model registry and versioning, packaging (ECR containers or model tar files), CI/CD and orchestration (SageMaker Pipelines, CodePipeline), and monitoring (CloudWatch, SageMaker Model Monitor). We follow a typical ML deployment journey: data and artifacts in Amazon S3 → model training in AWS SageMaker → model registration via Model Registry → packaging (ECR containers or model.tar.gz) → deployment (endpoints, batch, serverless) → monitoring and retraining automation.
The image outlines the stages of the machine learning deployment journey, including S3 data storage, SageMaker training, model registry, ECR packaging, and deployment.
Monitoring after deployment is essential. Typical AWS tools include:
  • AWS CloudWatch for infra, logs, and custom alarms.
  • SageMaker Model Monitor for data and prediction quality checks (data drift, feature distribution changes).
Tip: Treat the ML lifecycle as three versioned, monitored pillars — data, code, and infrastructure. Version artifacts, pipeline definitions, and infra templates to enable safe rollouts and fast rollback.

​
Why ML deployment is different from traditional software

  • Determinism: Traditional applications are typically deterministic — same inputs + same code = same outputs. ML models depend on data distributions and can produce different outcomes as data drifts.
  • Non-stationary inputs: ML systems must handle changes in input distributions (data drift, concept drift).
  • Continuous lifecycle: ML demands continuous monitoring, retraining, and reproducible pipelines to avoid performance regressions.
  • Artifact heterogeneity: Deployable ML artifacts can be serialized model files, container images, or compiled binaries (e.g., Neo-compiled artifacts).
The image is a comparison between traditional software and machine learning (ML) deployment, highlighting key differences such as deterministic outputs versus data dependency, and versioned code releases versus frequent retraining and monitoring.

​
Deployment patterns — what to choose and when

Choose a deployment pattern that matches latency, throughput, cost, and payload characteristics.
The image is an overview of deployment patterns, highlighting different request types including real-time, batch, asynchronous, and serverless inference using AWS SageMaker.
Warning: Using a real-time endpoint for large-batch scoring or highly variable traffic patterns can be cost-inefficient. Match the workload to the correct pattern and validate with load tests.

​
SageMaker deployment ecosystem — main components

  • Model Registry: central store for model versions, metadata, and approval workflows (stages: Staging, Production).
  • SageMaker model artifact: typically model.tar.gz (weights + inference code) or a container image in ECR.
  • Endpoints: support real-time, asynchronous, serverless, and multi-model hosting.
  • Multi-model endpoints: host multiple models in a single container for cost savings; consider cold-start and memory isolation trade-offs.
  • SageMaker Neo: compiles/optimizes models for edge devices and varied hardware targets.
The image is a flowchart depicting the SageMaker Deployment Ecosystem, with elements like S3 (data and artifacts), ECR (containers), Model Registry, VPC/IAM, and Endpoints connected to SageMaker.

​
Integration and packaging considerations

  • Amazon S3: store datasets and model artifacts (model.tar.gz). Apply lifecycle rules, enable KMS encryption, and limit access via IAM and VPC endpoints.
  • Amazon ECR: store custom inference containers. Use immutable tags or digests for reproducibility.
  • IAM and VPC: enforce least-privilege IAM roles for SageMaker, ECR, and S3. Use private VPC endpoints when required by compliance.
  • Packaging options:
    • model.tar.gz: use SageMaker built-in containers or small inference scripts.
    • Docker container (ECR): required for custom runtimes or unsupported frameworks.
  • Choosing pre-built containers vs BYOC (Bring Your Own Container): use pre-built for standard frameworks; use BYOC for custom dependencies, special runtimes, or system-level control.
The image illustrates the process of packaging model artifacts from S3 and ECR into SageMaker models, indicating either a direct model file or a custom container.

​
Deployment strategy patterns

Common rollout strategies for models deployed in production:
  • Blue-Green: run two independent environments (blue=current, green=new). Validate green and switch traffic when ready — simplifies rollback.
  • Canary: route a small percentage of traffic to the new model, monitor behavior, and progressively increase traffic if metrics are healthy.
  • Shadow (Mirroring): duplicate production requests to the candidate model without changing responses — good for validating correctness and performance under real traffic.
  • Rollback: keep models versioned in Model Registry and enable automated rollback in pipelines when monitoring detects regressions.
The image illustrates deployment strategies, showing a flowchart with client requests managed through an API Gateway/Load Balancer, splitting traffic between Blue (current) and Green (candidate) environments using Canary and Shadow flows.

​
CI/CD, orchestration, and retraining automation

A reliable ML CI/CD and automation pipeline ensures repeatability and safe rollouts. Typical flow:
  1. Code & pipeline definitions in source control (CodeCommit or GitHub).
  2. Build & test in CodeBuild (unit tests, model validation).
  3. Orchestrate with CodePipeline or third-party CI/CD.
  4. Trigger SageMaker Pipelines to run data preprocessing, training, evaluation, and registration.
  5. Register model to Model Registry and trigger deployment jobs (endpoint update, Canary, or Blue-Green).
  6. Automate retraining via EventBridge (on new data arrival), schedules, or data-change triggers.
Infrastructure as Code:
  • Use AWS CDK, CloudFormation, or Terraform to provision endpoints, autoscaling policies, IAM roles, and VPC configuration.
The image depicts a flowchart for MLOps and Deployment Automation, showcasing components like S3 Data storage, SageMaker Pipelines, Model Registry, Deployment (Endpoint/Batch), and Monitoring (CloudWatch/Model Monitor).

​
Common pitfalls and exam-focused reminders

  • No monitoring for drift: Without Model Monitor and CloudWatch alerts, model performance can silently degrade. Automate monitoring and alerting for prediction distributions and performance metrics.
  • Insufficient versioning or rollback plans: Always register models in Model Registry and include automated rollback steps in your pipelines.
  • Wrong deployment pattern: Matching workload-to-pattern matters — e.g., avoid using low-latency real-time endpoints for occasional, large-scale batch scoring.
  • Security and IAM misconfiguration: Apply least privilege, enable KMS encryption for S3/model artifacts, and restrict network access via VPC endpoints when required.
  • Cost traps: Persistent large instances or unused real-time endpoints drive costs. Consider autoscaling, multi-model endpoints, serverless hosting, and lifecycle policies to control spend.
The image lists four common deployment challenges and patterns: no monitoring, no versioning, wrong deployment pattern, and IAM misconfiguration, each with a brief explanation.

​
Summary — practical checklist

  • Package artifacts reproducibly (model.tar.gz or container images with immutable tags).
  • Register and stage models in Model Registry for approvals and traceability.
  • Select the correct deployment pattern (real-time, batch, async, serverless) based on latency, throughput, and cost.
  • Automate CI/CD and SageMaker Pipelines to make retraining and deployment repeatable.
  • Monitor data and predictions (Model Monitor + CloudWatch) and trigger retraining or rollback when needed.
  • Secure artifacts and infra using least-privilege IAM, KMS, and VPC controls.
This lesson covered the key considerations and AWS services for deploying and orchestrating machine learning models in production.

Watch Video