Skip to main content
This lesson summarizes the recommended patterns and operational practices for deploying and orchestrating machine-learning workflows on AWS. It covers the end-to-end journey from data storage and training to packaging, deployment, and production monitoring, while emphasizing security, cost, and operational trade-offs. A typical ML deployment journey on AWS follows these stages:
  • Store data and model artifacts in Amazon Simple Storage Service (Amazon S3).
  • Run training jobs with Amazon SageMaker using stored data.
  • Register and version the trained model in the SageMaker Model Registry.
  • Package the model and inference code into a container image and push it to Amazon Elastic Container Registry (ECR).
  • Choose the deployment pattern (real-time endpoint, batch transform, serverless, or asynchronous processing) and host the model.
  • Monitor production behavior with AWS CloudWatch and SageMaker Model Monitor to detect performance regressions or data/model drift.
For low-latency, on-demand predictions, a real-time SageMaker endpoint is typically appropriate.
The image illustrates "The ML Deployment Journey" using AWS services, including S3, SageMaker Training, Model Registry, ECR (for packaging), Deployment, and CloudWatch/Model Monitor.
Deployment pattern selection depends on request characteristics, throughput, latency tolerance, payload size, operational complexity, and cost. Common patterns:
  • Real-time (low-latency, synchronous): online endpoints for single-request predictions.
  • Batch (asynchronous bulk): transform jobs for large datasets with no strict latency constraints.
  • Async (long-running or large payloads): de-coupled request/response using queues or asynchronous endpoints to avoid timeouts.
  • Serverless (bursty or low-traffic): cost-effective for infrequent traffic with automatic scaling.
Use the following table to compare patterns at a glance:
The image illustrates deployment patterns categorized into Real-Time, Batch, Async, and Serverless, highlighting different request types. It also considers factors like latency, throughput, payload size, and cost.
Container strategy and orchestration choices Container options range from managed runtimes and pre-built SageMaker containers to BYOC (bring-your-own-container) and fully self-managed orchestration (ECS/EKS). The trade-off is simple: more control increases operational overhead (security, updates, image size, tuning). Key container best practices:
  • Apply least-privilege IAM roles for SageMaker, ECR, S3, and other services.
  • Encrypt artifacts and images using AWS KMS.
  • Scan container images for vulnerabilities and maintain a Software Bill of Materials (SBOM).
  • Use VPC endpoints and run endpoints inside VPCs when private network access is required.
  • Choose smaller base images and managed runtimes to reduce cold-starts and maintenance.
Container choice affects security, cost, and operational complexity. Prefer managed runtimes when possible and minimize image surface area to reduce vulnerabilities and cold starts.
The image outlines "Containers – Security Best Practices" focusing on IAM, KMS, image scanning, and VPC endpoints, with recommendations for least-privilege roles, encryption, and private network access.
Edge deployment with SageMaker Neo For on-device inference, SageMaker Neo compiles and optimizes trained models for specific hardware targets. Typical workflow:
  • Start with a trained model artifact from SageMaker or another training environment.
  • Use SageMaker Neo to compile and optimize the model for the target edge device.
  • Deploy the compiled artifact to the device.
Benefits include reduced latency, lower power usage, and improved runtime performance on constrained hardware.
The image is a diagram titled "Edge Deployment With SageMaker Neo," outlining steps from a trained model to deployment on an edge device, highlighting benefits like reduced latency, lower power usage, and improved performance.
Infrastructure as Code (IaC) for ML When you apply IaC to ML, evaluate these four categories:
Avoid anti-patterns such as embedded secrets, overly broad IAM permissions, and manual configuration changes that bypass IaC. These lead to security and reliability problems.
The image is a diagram titled "IAC for ML" outlining categories and considerations for machine learning infrastructure as code, including tools, security, operational excellence, and anti-patterns to avoid. Each category lists relevant items like CloudFormation, IAM least privilege, cost optimization, and avoiding hardcoding secrets.
CI/CD and MLOps automation A robust ML CI/CD pipeline on AWS commonly uses:
  • AWS CodePipeline to orchestrate end-to-end automation and trigger runs on code or data changes.
  • AWS CodeBuild to compile, run unit tests, build artifacts, and package models.
  • AWS CodeDeploy or other deployment mechanisms to automate rollouts.
  • Amazon SageMaker for training jobs, model registration, and serving.
These services enable repeatable, auditable automation that integrates build, test, and deploy stages for MLOps workflows. Recommended security layers for pipelines:
  • Identity and access control with scoped IAM roles and policies.
  • Encryption of code, data, and model binaries using KMS.
  • Network isolation using VPCs and private endpoints.
  • Audit logging and monitoring using AWS CloudTrail and CloudWatch.
The image illustrates the process of implementing machine learning CI/CD using AWS services: CodePipeline, CodeBuild, CodeDeploy, and SageMaker. It shows the flow from building and testing to deployment and managing ML jobs.
Testing and secure environments Testing should occur in protected, representative environments. Key practices:
  • Apply strict IAM scoping and role separation for test workloads.
  • Encrypt test data at rest (KMS) and in transit (TLS).
  • Use data masking, pseudonymization, or synthetic datasets to avoid exposing production-sensitive information.
  • Automate test environment provisioning via IaC to avoid configuration drift.
The image details components of security in machine learning testing, including IAM roles and policies, encryption, and data masking, all contributing to a secure test environment.
Key takeaways
  • ML deployment is a continuous lifecycle: train → register → package → deploy → monitor.
  • Pick the deployment pattern (real-time, batch, async, serverless, edge) based on latency, throughput, payload size, cost, and operational constraints.
  • Use containers and ECR for consistent, portable deployments; prefer small, scanned images and managed runtimes to reduce operational overhead.
  • Build security around least-privilege IAM, KMS-based encryption, VPC isolation, and centralized auditing/logging.
  • Automate delivery with CI/CD (CodePipeline, CodeBuild, CodeDeploy and SageMaker) for repeatability and traceability.
  • Optimize for edge with SageMaker Neo when on-device performance and power efficiency are priorities.
  • Avoid anti-patterns: do not hard-code secrets, do not use wildcard IAM permissions, prevent configuration drift, and include rollback and disaster recovery plans.
A secure, resilient ML system combines strong identity controls, encryption, private networking, automated pipelines, and continuous monitoring to maintain model performance and compliance through production.

Watch Video