Cost Analysis and Optimization for ML Workloads

This lesson explains how to analyze and optimize costs for machine learning workloads on AWS. Effective ML cost management requires instrumentation (collecting usage data), analysis (attributing spend to workloads), and operational guardrails (automation, rightsizing, and policy controls). Below are the key AWS services and patterns to use when building a cost-aware ML platform.

AWS Service	Use case for ML workloads	Example
Cost Explorer	Visualize spending, trends, and tag-based allocation	AWS Cost Explorer
Amazon SageMaker	Training & inference with spot, managed endpoints, and multi-instance training	AWS SageMaker
Amazon S3	Tiered storage & lifecycle policies for datasets and model artifacts	Amazon S3
AWS CloudWatch	Monitor utilization, billing metrics, and alarms	AWS CloudWatch
Compute Optimizer	Recommendations to right-size instances	Compute Optimizer
Trusted Advisor	Best-practice checks including cost-savings opportunities	Trusted Advisor
IAM	Enforce least-privilege to prevent unauthorized/spurious spending	AWS IAM

A presentation slide titled "Cost Analysis and Optimization – AWS Tools" showing icons and labels for AWS services such as Cost Explorer, SageMaker, S3, CloudWatch, Compute Optimizer, Trusted Advisor, and IAM.

1) Start with cost attribution: tags and Cost Explorer

Begin by collecting and tagging ML resources consistently (for example MLWorkload=FraudDetection). Feeding those tags to Cost Explorer enables accurate, workload-level billing analysis and trend detection.

A dark presentation slide titled "Tools for ML Workloads – AWS Cost Explorer" showing a three-step flow: ML Workloads → AWS Cost Explorer → Billing Insights, each represented by an icon.

Example: query Cost Explorer for the total cost in June 2025, grouped and filtered by tag MLWorkload=FraudDetection. Use the ce boto3 client and ensure the account has ce:GetCostAndUsage permission.

# Python example: Query Cost Explorer for tagged ML workload cost
import boto3

ce = boto3.client('ce', region_name='us-east-1')

response = ce.get_cost_and_usage(
    TimePeriod={
        'Start': '2025-06-01',
        'End': '2025-07-01'
    },
    Granularity='MONTHLY',
    Metrics=['UnblendedCost'],
    GroupBy=[
        {
            'Type': 'TAG',
            'Key': 'MLWorkload'
        }
    ],
    Filter={
        'Tags': {
            'Key': 'MLWorkload',
            'Values': ['FraudDetection']
        }
    }
)

print(response['ResultsByTime'])

Tagging resources consistently (for example MLWorkload=FraudDetection) is critical for attributing costs correctly across teams and workloads. Make tagging part of CI/CD and provisioning templates.

2) Reduce compute costs: SageMaker Spot, managed training, and checkpointing

Compute (training and inference) typically drives most ML cloud spend. Consider these strategies:

Use SageMaker Managed Spot Training to leverage excess EC2 capacity at large discounts.
Enable checkpointing so interrupted Spot instances resume without losing progress.
Use multi-instance distributed training only when it lowers wall-clock time sufficiently to justify extra instances.
Combine Spot with Reserved or Savings Plans for predictable baseline workloads.

A presentation slide titled "Optimizing Compute Costs With SageMaker" showing three numbered points: on-demand training is costly, SageMaker Spot instances can cut costs up to 70%, and checkpointing should be enabled because Spot instances can be interrupted.

Spot instances can be interrupted with little notice. Always design training jobs to checkpoint model state and resume training to avoid lost progress and wasted resource time.

Example: create a SageMaker training job with Managed Spot Training. Note EnableManagedSpotTraining=True and MaxWaitTimeInSeconds to let the job wait for Spot capacity.

# Python example: Create a SageMaker training job with Managed Spot Training
import boto3

sm = boto3.client('sagemaker', region_name='us-east-1')

sm.create_training_job(
    TrainingJobName='fraud-training-job',
    AlgorithmSpecification={
        'TrainingImage': '123456789012.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:latest',
        'TrainingInputMode': 'File'
    },
    RoleArn='arn:aws:iam::123456789012:role/SageMakerRole',
    InputDataConfig=[
        {
            'ChannelName': 'train',
            'DataSource': {
                'S3DataSource': {
                    'S3DataType': 'S3Prefix',
                    'S3Uri': 's3://ml-bucket/train-data/',
                    'S3DataDistributionType': 'FullyReplicated'
                }
            }
        }
    ],
    OutputDataConfig={
        'S3OutputPath': 's3://ml-bucket/output/'
    },
    ResourceConfig={
        'InstanceType': 'ml.p3.2xlarge',
        'InstanceCount': 1,
        'VolumeSizeInGB': 50
    },
    EnableManagedSpotTraining=True,
    StoppingCondition={
        'MaxRuntimeInSeconds': 86400,
        'MaxWaitTimeInSeconds': 86400
    }
)

3) Optimize storage costs with S3 lifecycle policies

Storage for training data, intermediate artifacts, and model checkpoints can be expensive if left in hot storage. Apply lifecycle policies that automatically move objects to lower-cost tiers as access patterns age. A common lifecycle progression: S3 Standard → Intelligent‑Tiering → Standard‑IA → Glacier / Glacier Deep Archive.

A dark-themed flowchart titled "Optimizing Storage Costs With S3" showing an S3 bucket icon and a lifecycle policy that moves data through tiers: S3 Standard (30 days), S3 Intelligent‑Tiering (60 days), S3 Standard‑IA (90 days) and finally S3 Glacier Deep Archive.

Example lifecycle rule: objects under prefix ml-data/ are transitioned to Glacier after 30 days. Adjust StorageClass to DEEP_ARCHIVE if you require Glacier Deep Archive.

{
  "Rules": [
    {
      "ID": "ArchiveMLData",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "ml-data/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

Best practices for S3 and ML:

Store frequently used datasets in Standard or Intelligent‑Tiering.
Use object lifecycle rules on experiment/temporary folders.
Use S3 object locking/versioning policies when you need retention guarantees but still want to tier storage.

4) Monitor costs and tune monitoring to avoid runaway monitoring spend

Monitoring itself can generate recurring costs (high-cardinality metrics, long retention, noisy alarms). Review CloudWatch usage and optimize:

Reduce retention windows for rarely used metrics.
Avoid high-cardinality dimensions unless essential.
Aggregate metrics where possible and tune alarm thresholds to reduce false positives.

A dark-themed slide titled "Optimizing Monitoring Costs With CloudWatch" showing a four-step cost optimization workflow. The steps are CloudWatch Monitoring Setup → Analyze Costs → Identify High-Cost Metrics and Alarms → Apply Monitoring Optimizations, with tips like reducing data retention, adjusting alarm thresholds, and selective metric collection.

A practical guardrail is a billing alarm for estimated charges. The JSON below shows a typical CloudWatch alarm targeted at the EstimatedCharges metric scoped to the AmazonCloudWatch service and configured to notify an SNS topic.

{
  "AlarmName": "MLMonitoringCostOverrun",
  "MetricName": "EstimatedCharges",
  "Namespace": "AWS/Billing",
  "Threshold": 1000.0,
  "ComparisonOperator": "GreaterThanThreshold",
  "EvaluationPeriods": 1,
  "Period": 21600,
  "Dimensions": [
    {
      "Name": "ServiceName",
      "Value": "AmazonCloudWatch"
    }
  ],
  "AlarmActions": [
    "arn:aws:sns:us-east-1:123456789012:AlertTopic"
  ]
}

Table: Monitoring optimizations

Area	Action
Metrics	Aggregate or sample high-cardinality metrics; avoid thousands of unique dimensions
Retention	Lower retention for infrequently accessed metrics
Alerts	Use composite alarms or anomaly detection to reduce noise
Costs	Monitor `AWS/Billing` metrics and create SNS-driven notifications

5) Scale inference efficiently with Auto Scaling

For variable inference traffic, use Auto Scaling (EC2 Auto Scaling groups, ECS/EKS autoscaling, or SageMaker endpoint autoscaling) to match capacity to demand. Scale both horizontally and vertically where appropriate, and use predictive scaling if your traffic has predictable patterns.

A slide titled "Auto-Scaling for Cost Efficiency" showing a three-step flow: Incoming Traffic → Auto Scaling Group → Right-Sized Instances.

6) Rightsizing and governance

Right-sizing is iterative:

Run representative training and inference workloads.
Use Compute Optimizer to analyze utilization and recommend instance sizes.
Review Trusted Advisor for cost-related best practices.
Apply changes gradually and measure performance vs. cost tradeoffs.

Security & governance: enforce least privilege with IAM so only authorized principals can create compute or storage resources or view billing data. Below is a minimal example policy that grants permissions for cost monitoring and core ML actions — adjust ARNs and resource names for your account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ce:GetCostAndUsage",
        "sagemaker:CreateTrainingJob",
        "s3:PutObject",
        "kms:Encrypt"
      ],
      "Resource": [
        "*",
        "arn:aws:sagemaker:us-east-1:123456789012:training-job/*",
        "arn:aws:s3:::ml-bucket/*",
        "arn:aws:kms:us-east-1:123456789012:key/*"
      ]
    }
  ]
}

Common anti-patterns that drive unnecessary spend

Relying only on on-demand pricing (missing Spot, Reserved, or Savings Plan optimizations).
Not applying S3 lifecycle rules (paying premium for cold data).
Ignoring billing alarms (delaying detection of overruns).
Not right-sizing workloads (running idle or overpowered instances).
Allowing broad IAM permissions that enable accidental resource creation.

Immediate action checklist (practical next steps)

Priority	Action
High	Use Cost Explorer with consistent tags to identify high-cost ML workloads
High	Implement SageMaker Managed Spot Training + checkpointing for large training jobs
Medium	Add S3 lifecycle rules to tier historical datasets and model artifacts
Medium	Create CloudWatch billing alarms and review CloudWatch metric retention
Low	Run Compute Optimizer and Trusted Advisor checks and apply rightsizing recommendations

Links and references

Takeaway: instrument cost data, tag resources consistently, combine Spot and reserved-capacity strategies where appropriate, automate data-tiering for S3, monitor both application and billing metrics, and enforce least-privilege to avoid accidental spend.

Prerequisites

Data Preparation for Machine Learning (ML)

ML Model Development

Deployment and Orchestration of ML Workflows

ML Solution Monitoring, Maintenance, and Security

Bringing it all together

Cost Analysis and Optimization for ML Workloads

1) Start with cost attribution: tags and Cost Explorer

2) Reduce compute costs: SageMaker Spot, managed training, and checkpointing

3) Optimize storage costs with S3 lifecycle policies

4) Monitor costs and tune monitoring to avoid runaway monitoring spend

5) Scale inference efficiently with Auto Scaling

6) Rightsizing and governance

Common anti-patterns that drive unnecessary spend

Immediate action checklist (practical next steps)

Links and references

Watch Video

​1) Start with cost attribution: tags and Cost Explorer

​2) Reduce compute costs: SageMaker Spot, managed training, and checkpointing

​3) Optimize storage costs with S3 lifecycle policies

​4) Monitor costs and tune monitoring to avoid runaway monitoring spend

​5) Scale inference efficiently with Auto Scaling

​6) Rightsizing and governance

​Common anti-patterns that drive unnecessary spend

​Immediate action checklist (practical next steps)

​Links and references

Watch Video

1) Start with cost attribution: tags and Cost Explorer

2) Reduce compute costs: SageMaker Spot, managed training, and checkpointing

3) Optimize storage costs with S3 lifecycle policies

4) Monitor costs and tune monitoring to avoid runaway monitoring spend

5) Scale inference efficiently with Auto Scaling

6) Rightsizing and governance

Common anti-patterns that drive unnecessary spend

Immediate action checklist (practical next steps)

Links and references