Skip to main content
In this lesson we’ll apply DevOps principles to machine learning by building an automated MLOps CI/CD pipeline on AWS. The pipeline uses AWS CodePipeline for orchestration, CodeBuild for building and testing, and CodeDeploy for controlled rollouts where appropriate. Amazon SageMaker is the primary runtime for training, model registry, and serving endpoints. Key goals:
  • Automate reproducible model training and delivery.
  • Validate models with automated tests and metrics gates.
  • Deploy models safely using traffic-shifting or rollout strategies.
  • Secure the pipeline and artifacts using least-privilege IAM, KMS, and network controls.
High-level workflow:
  • CodePipeline orchestrates stages from source to deployment.
  • CodeBuild installs dependencies, runs tests, and creates artifacts or container images.
  • SageMaker runs training jobs, registers models, and hosts endpoints.
  • CodeDeploy provides controlled rollouts for compute targets it supports (ECS, Lambda, EC2).
The following diagram shows the overall concept of an ML CI/CD pipeline using AWS developer tools.
The image is a flowchart demonstrating the implementation of ML CI/CD using AWS services: CodePipeline, CodeBuild, CodeDeploy, and Amazon SageMaker.
Why automate ML with CI/CD?
  • Manual workflows are error-prone, slow to iterate, and hard to reproduce.
  • CI/CD enforces automated builds, tests, and structured deployments so results are repeatable and delivery is faster.
Compare manual ML workflows with CI/CD automation:
The image compares the use of AWS Code Services for ML automation with and without CI/CD. It highlights issues like manual steps and frequent errors without CI/CD and benefits of automation with it.
Core pipeline orchestration
  • CodePipeline coordinates stages such as Source → Build → Train → Validate → Deploy → Monitor.
  • It triggers CodeBuild to run unit tests, data checks, packaging, and container image builds.
  • Pipeline stages call SageMaker to run training jobs and register models in a model registry.
  • For serving, prefer SageMaker native endpoint features; for other compute targets use CodeDeploy for blue/green or canary rollouts.
The architecture below shows how CodePipeline integrates CodeBuild, CodeDeploy, and SageMaker.
The image illustrates how AWS services support ML CI/CD, featuring CodeDeploy, CodeBuild, SageMaker, and CodePipeline as key components in the process.
End-to-end flow example
  1. Developer or data engineer pushes code or data to the source repo.
  2. CodePipeline detects the commit and starts the pipeline.
  3. CodeBuild bundles dependencies, runs tests, and produces an artifact or container image.
  4. The pipeline triggers a SageMaker training job or uses the built image for training.
  5. After validation, the model is registered and deployed to an endpoint or compute target.
  6. Monitoring and automated rollback gates ensure safety.
Visual flow of this sequence:
The image illustrates how AWS services support ML CI/CD, featuring CodeDeploy, CodeBuild, SageMaker, and CodePipeline as key components in the process.
CodeBuild: packaging, testing, and artifacts
  • Install dependencies, run unit/integration tests, and execute data validation.
  • Create model packages or build container images pushed to ECR.
  • Produce artifacts consumed by SageMaker training or later deployment stages.
The image illustrates a workflow for automating model packaging using AWS CodeBuild, starting with source code and leading to build artifacts or an ML container.
Deployment strategies for ML models
  • Blue/Green: provision a parallel environment and switch traffic once tests pass.
  • Canary: route a small % of traffic to the new model, then increase if metrics are healthy.
  • Shadow Testing: mirror production traffic to the new model without affecting user responses.
Use SageMaker variant-weighting and traffic-shifting for endpoint deployments, or orchestrate the rollout from CodePipeline. For other compute types, leverage CodeDeploy’s strategies.
The image is a flowchart showing how AWS CodeDeploy manages model releases using deployment strategies, specifically Blue/Green and Canary.
Orchestrating stages and artifacts A well-structured pipeline produces artifacts and test outputs at each stage so downstream stages can validate automatically and make branching decisions (promote, rollback, or notify). Security practices for ML CI/CD
  • Use least-privilege IAM roles for CodePipeline, CodeBuild, SageMaker, and CodeDeploy.
  • Encrypt artifacts and model packages with AWS KMS.
  • Run SageMaker (and other compute) in a private VPC and restrict network access.
The image outlines steps for securing ML CI/CD on AWS, including CI/CD services, IAM roles with KMS encryption, and secure SageMaker deployment using a private VPC.
Choosing the right rollout from your model registry
The image outlines safe deployment strategies for ML models, including Blue/Green, Canary, and Shadow Testing, starting from a Model Registry.
Automated validation and gating
  • Build automated checks into the pipeline: unit tests, data schema validations, data drift checks, and model evaluation metrics compared against baselines.
  • Only promote models that meet the acceptance criteria; on failure, the pipeline should stop and notify the team.
The image is a flowchart showing automated testing and validation in a CI/CD pipeline. It outlines steps from the pipeline to automated tests, leading to deployment if tests pass, or stopping and fixing if tests fail.
Common CI/CD anti-patterns to avoid
  • Skipping automated tests to save time.
  • Hardcoding secrets into pipeline scripts or artifacts.
  • Failing to version control code and model artifacts, preventing reliable rollbacks.
The image lists common CI/CD pitfalls: "Skipping Tests," "Hardcoding Secrets," and "No Version Control." It includes icons for each issue and suggests how to avoid them.
Avoid these pitfalls: always include automated tests, store secrets in a secrets manager (for example, AWS Secrets Manager (https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) or Systems Manager Parameter Store), and keep all code and model artifacts in versioned storage.
Quick reference: AWS components and responsibilities Recommendations and best practices
  • Automate packaging, validation, and deployment steps; ensure observability and monitoring at each stage.
  • Enforce security using IAM least-privilege, KMS encryption, and VPC isolation for compute.
  • Gate deployments with robust automated tests and model-metric thresholds.
  • Use blue/green, canary, or shadow testing to minimize risk during rollouts.
  • Maintain versioned storage for code, artifacts, and models to enable safe rollbacks.
Automate packaging, validation, and deployment as part of your pipeline and monitor each stage. This reduces risk, improves reproducibility, and enables faster, safer model delivery.
Links and references By combining CodePipeline, CodeBuild, SageMaker, and CodeDeploy with secure practices and automated validation, you can build an MLOps pipeline that enables repeatable training, safe deployments, and rapid model iteration.

Watch Video