Monitoring and Observability Part 2

Infrastructure metrics tell part of the story, but to truly understand model behavior — bias, feature importance, or data quality issues — you need higher-level monitoring that examines inputs, predictions, and outcomes. Amazon SageMaker Model Monitor is a native capability that helps you detect drift and quality regressions. It integrates with SageMaker endpoints, S3, CloudWatch, and scheduled processing jobs to provide automated baselining, continuous monitoring, and HTML reports. In this article we cover the four main monitoring types available in SageMaker Model Monitor:

Data Quality Monitoring
Model Quality Monitoring
Bias Drift Monitoring
Feature Attribution Drift Monitoring

We’ll show how to create baselines for each type, enable capture of inference requests, schedule recurring monitoring jobs, and interpret the resulting reports. At a high level, these four monitor types are:

Note: each monitor type requires its own baseline generation job, endpoint capture configuration, and scheduled monitoring job.

Data Quality Monitoring

Data quality monitoring verifies that inference input data remains statistically similar to the training data. The goal is to detect schema changes, missing fields, invalid values, and distribution drift by comparing captured inference request samples against a training-time baseline. To generate a baseline, use DefaultModelMonitor.suggest_baseline. That launches a processing job which computes statistics and constraints and writes them to S3.

from sagemaker import get_execution_role
from sagemaker.model_monitor import DefaultModelMonitor

role = get_execution_role()

baseline_job = DefaultModelMonitor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge"
)

baseline_results_s3_uri = "s3://your-bucket/baseline-results/"
baseline_data_s3_uri = "s3://your-bucket/baseline-data/inference_data.csv"

baseline_job.suggest_baseline(
    baseline_dataset=baseline_data_s3_uri,
    dataset_format={"csv": {"header": False}},  # Adjust format based on your data
    output_s3_uri=baseline_results_s3_uri,
    wait=True
)

print("Baseline job completed. Results stored in:", baseline_results_s3_uri)

After creating the baseline, enable data capture on your SageMaker endpoint so a sample of inference requests is saved to S3. Once captured, schedule a processing job that compares the incoming distribution to the baseline and writes reports to S3 and logs to CloudWatch.

A diagram titled "Workflow: Data Quality Monitoring" showing an AWS SageMaker-based pipeline with training data, a Model Monitor data quality baseline job, S3 buckets for baselines and logged inference requests, and a managed ML instance running a SageMaker frontend, inference handler, and model. It also shows scheduled monitoring, an S3 bucket sink, and CloudWatch integration.

Example: create and schedule a data quality monitoring job that compares captured requests with baseline statistics and writes results to S3/CloudWatch.

from sagemaker.model_monitor import ModelMonitor

monitoring_schedule_name = "data-quality-monitoring-job"

monitor = ModelMonitor(
    role=role,
    image_uri=baseline_job.image_uri,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    volume_size_in_gb=20,
    max_runtime_in_seconds=3600,
)

monitor.create_monitoring_schedule(
    monitoring_schedule_name=monitoring_schedule_name,
    endpoint_input=endpoint_name,
    output_s3_uri="s3://your-bucket/monitoring-results/",
    statistics_s3_uri=f"{baseline_results_s3_uri}statistics.json",
    constraints_s3_uri=f"{baseline_results_s3_uri}constraints.json",
    schedule_cron_expression="cron(0 * * * ? *)",  # Runs every hour
)

print("Monitoring schedule created:", monitoring_schedule_name)

Model Quality Monitoring

Model quality monitoring looks for concept drift: when predictions no longer align with real-world outcomes. This requires ground-truth labels that you can pair with captured predictions. Note that ground truth can be delayed in many production scenarios (for example churn, fraud, or medical outcomes), so design your cadence and retention policies accordingly.

Ground truth labels are required for model quality monitoring. There is often a time lag between making a prediction and observing the true label — design your monitoring cadence and retention policies to account for this delay.

Create a model quality baseline that captures initial alignment between predictions and ground-truth. Use ModelQualityMonitor.suggest_baseline and specify the problem type: Regression, BinaryClassification, or MulticlassClassification.

from sagemaker.model_monitor import ModelQualityMonitor

baseline_results_s3_uri = "s3://your-bucket/baseline-results/model-quality/"
baseline_data_s3_uri = "s3://your-bucket/baseline-data/inference_outputs.csv"
ground_truth_s3_uri = "s3://your-bucket/baseline-data/ground_truth_labels.csv"

model_quality_monitor = ModelQualityMonitor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge"
)

model_quality_monitor.suggest_baseline(
    baseline_dataset=baseline_data_s3_uri,
    problem_type="Regression",  # Use "BinaryClassification" or "MulticlassClassification" as needed
    ground_truth_dataset=ground_truth_s3_uri,
    output_s3_uri=baseline_results_s3_uri,
    wait=True
)

print("Model Quality Baseline Completed. Results stored at:", baseline_results_s3_uri)

Ground-truth files should match inferenceId values to actual labels so the monitor can pair predictions with outcomes. Example ground-truth JSON lines:

{"inferenceId": "abc123", "actualLabel": 1}
{"inferenceId": "def456", "actualLabel": 0}
{"inferenceId": "ghi789", "actualLabel": 1}

After enabling capture on your endpoint and collecting ground truth in S3, schedule a model quality monitoring job. The monitor will compute metrics such as accuracy, F1, MSE, and compare them to thresholds to produce an HTML performance report.

from sagemaker.model_monitor import ModelQualityMonitor

model_monitoring_schedule_name = "model-quality-monitor"

model_monitor = ModelQualityMonitor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    volume_size_in_gb=20,
    max_runtime_in_seconds=3600,
)

model_monitor.create_monitoring_schedule(
    monitoring_schedule_name=model_monitoring_schedule_name,
    endpoint_input=endpoint_name,
    ground_truth_input="s3://your-bucket/ground-truth/",
    problem_type="Regression",  # Change for classification
    output_s3_uri="s3://your-bucket/model-quality-results/",
    statistics_s3_uri=f"{baseline_results_s3_uri}statistics.json",
    constraints_s3_uri=f"{baseline_results_s3_uri}constraints.json",
    schedule_cron_expression="cron(30 * * * ? *)",  # Runs every hour at minute 30
)

print("Model Quality Monitoring Schedule Created:", model_monitoring_schedule_name)

A slide titled "Workflow: Model Quality Monitoring" showing four numbered steps. It lists: track prediction accuracy, compute ML metrics (MSE, accuracy, recall, F1), raise alerts when accuracy falls, and decide when to retrain the model.

Bias Drift Monitoring

Bias drift monitoring checks whether model predictions show unfair behavior across sensitive groups (for example, gender, race, or age). You define sensitive features and the monitor computes fairness metrics (disparate impact, group-wise performance, etc.) to detect bias shifts. Bias monitoring also requires its own baseline derived from historical or training data. If you use multiple monitors, create a separate baseline for each monitor type.

Example: create a bias baseline then schedule a bias monitoring job. Including ground truth improves fidelity of fairness calculations.

from sagemaker.model_monitor import BiasMonitor
from sagemaker import get_execution_role

role = get_execution_role()

bias_monitor = BiasMonitor(role=role)

bias_monitor.suggest_baseline(
    baseline_dataset='s3://path-to-your-baseline-dataset.csv',  # Historical data
    output_s3_uri='s3://path-to-baseline-output',
    sensitive_features=['gender', 'age'],
    wait=True
)

Enable data capture on your endpoint, prepare captured predictions and (optionally) ground-truth labels, then create a scheduled bias monitoring job:

bias_monitor.create_monitoring_schedule(
    monitoring_schedule_name="bias-monitoring-schedule",
    endpoint_input="s3://path-to-captured-inference-data/",
    ground_truth_input="s3://path-to-ground-truth-data/",
    baseline_input="s3://path-to-baseline-data/",
    sensitive_features=['gender', 'age'],
    output_s3_uri="s3://path-to-monitoring-results/",
    schedule_cron_expression="cron(30 * * * ? *)"  # Runs every hour at minute 30
)

The bias report will include group-wise metrics and highlight potential fairness issues across the specified sensitive features.

A presentation slide titled "Workflow: Model Quality Monitoring" with three numbered panels outlining steps: 01 Unique ID, 02 Matching, and 03 File. Each panel briefly explains using an inferenceId to match predictions, including it in requests/ground truth, and storing results in JSON with inferenceId and actualLabel.

Feature Attribution Drift Monitoring

Feature attribution drift monitoring detects changes in which features the model relies on to make predictions. Typically this uses attribution methods (e.g., SHAP) to compute per-feature importance for inference records, store the attribution values as a baseline, and compare live attributions to that baseline. Significant shifts in attribution can indicate model degradation or a change in the underlying data-generating process. For example, if square footage initially had the highest attribution for house price predictions but location later becomes the dominant factor, attribution drift would surface that shift so you can investigate feature engineering, data collection, or retraining.

A slide titled "Workflow: Feature Attribution Drift Monitor" showing a Feature Attribution Drift Report that flags drift (Square Footage attribution ↓16.67%, Location attribution ↑66.67%) with a "Drift Detected!" alert. Next to it are recommended actions: Investigate data shift, Retrain the model with updated data, and Enhance feature selection.

The attribution baseline job computes initial attribution values and stores both statistics and attribution outputs (CSV) in S3.

A diagram titled "Workflow: Feature Attribution Drift Monitor" showing training data flowing into a Model Monitor Attribution Drift Baseline Job that writes outputs to a storage bucket. The bucket contains files named statistics.json, constraints.json, attribution_data.csv, and model_metadata.json.

Example: compute a feature-attribution baseline, enable capture, then schedule the attribution drift monitor.

from sagemaker.model_monitor import FeatureAttributionMonitor
from sagemaker import get_execution_role

role = get_execution_role()

feature_attribution_monitor = FeatureAttributionMonitor(role=role)

# Create a processing job to compute feature attribution and store the baseline in S3
baseline_input = "s3://path-to-baseline-inference-data/"
output_s3_uri = "s3://path-to-store-baseline-output/"
feature_attribution_monitor.create_processing_job(
    baseline_input=baseline_input,
    output_s3_uri=output_s3_uri,
    baseline_name="feature-attribution-baseline"
)

After the baseline generation and endpoint capture are in place, schedule the monitoring job that compares live attributions to the baseline:

feature_attribution_monitor.create_monitoring_schedule(
    monitoring_schedule_name="feature-attribution-drift-monitoring-schedule",
    endpoint_input="s3://path-to-captured-inference-data/",
    baseline_input="s3://path-to-baseline-data/",
    output_s3_uri="s3://path-to-monitoring-results/",
    schedule_cron_expression="cron(0 12 * * ? *)"  # Runs daily at 12:00 PM UTC
)

Comparison Summary Table

Monitor Type	Purpose	Required Inputs	Typical Outputs / Metrics
Data Quality Monitoring	Detect schema changes, missing fields, and distribution drift	Training baseline, captured inference inputs (S3)	`statistics.json`, `constraints.json`, HTML reports, CloudWatch logs
Model Quality Monitoring	Detect concept drift vs. ground truth	Captured predictions, ground-truth labels, baseline	Accuracy, F1, MSE, HTML performance reports, alerts
Bias Drift Monitoring	Detect fairness issues across sensitive groups	Baseline with sensitive features, captured data, (optional) ground truth	Group-wise metrics, disparate impact, bias reports
Feature Attribution Drift Monitoring	Detect changes in feature importance	Attribution baseline (e.g., SHAP), captured attributions	Attribution CSV, drift alerts, recommendations to retrain or reengineer features

Wrap-up

SageMaker Model Monitor provides a robust set of tools to detect data drift, concept drift, fairness regressions, and changes in feature importance. Key operational steps for each monitor type are:

Generate a baseline from historical/training data.
Enable endpoint data capture so inference requests (and optionally responses) are written to S3.
Collect ground truth labels when required (for model quality and recommended for bias).
Schedule recurring monitoring jobs that compare live data/attributions against baselines.
Inspect HTML reports and CloudWatch logs to drive remediation (retrain, change features, or collect more representative data).

A presentation slide titled "Summary" showing a numbered list of monitoring types. Each item briefly describes Data Quality, Model Quality, Bias, and Feature Attribution monitoring with example checks like schema/data drift, accuracy (MSE/F1), fairness comparisons, and feature influence.

That wraps up this lesson on monitoring with SageMaker Model Monitor. In a follow-up guide we’ll demonstrate an end-to-end example that wire-taps an endpoint, stores captured data in S3, pushes ground-truth labels, and runs scheduled monitors to trigger CloudWatch alerts and HTML reports.

Machine Learning Prerequisites

SageMaker Introduction

SageMaker User Interface

Persona SageMaker Activities - Data Engineer

Persona SageMaker Activities - Data Scientist

Persona SageMaker Activities - MLOps Engineer

Wrap Up

Course Introduction

Monitoring and Observability Part 2

Data Quality Monitoring

Model Quality Monitoring

Bias Drift Monitoring

Feature Attribution Drift Monitoring

Comparison Summary Table

Wrap-up

Watch Video

​Data Quality Monitoring

​Model Quality Monitoring

​Bias Drift Monitoring

​Feature Attribution Drift Monitoring

​Comparison Summary Table

​Wrap-up

Watch Video

Data Quality Monitoring

Model Quality Monitoring

Bias Drift Monitoring

Feature Attribution Drift Monitoring

Comparison Summary Table

Wrap-up