Introduction to Experiment Management

• The problem of tracking iterative experiments in SageMaker Studio.
• How SageMaker Experiments helps with tracking and reproducibility.
• A code-first workflow using the SageMaker SDK.
• How to capture and compare runs to reproduce results.

​

The problem: many iterative experiments, poor traceability

• Model type (e.g., RandomForest, XGBoost)
• Hyperparameters and optimization schedules
• Feature engineering or preprocessing pipelines
• Dataset versions or label corrections

• Experiment 1: RandomForest with scaling and PCA
• Experiment 2: XGBoost without scaling with a different learning rate
• Experiment 3: XGBoost with alternate hyperparameters and feature set

​

Solution: experiment management with SageMaker Experiments

• Automatically capture algorithm/container versions, hyperparameters, dataset identifiers, and training artifacts.
• Compare metrics and visualize trends across runs.
• Reproduce successful runs by restoring the recorded inputs and hyperparameters.
• Filter and analyze many runs (for example, show only runs with RMSE < X).

​

SageMaker Experiments: structure and terminology

​

Using the SDK — a code-first workflow

1. Create an Experiment and Trial.
2. Open a Run context.
3. Log parameters and metrics.
4. Execute a training job inside the Run so that SageMaker automatically associates the training job as a trial component.

# Example: Using SageMaker Experiments with an Estimator (XGBoost)
import sagemaker
from sagemaker.experiments.run import Run
from sagemaker.experiments.experiment import Experiment
from sagemaker.experiments.trial import Trial
from sagemaker.estimator import Estimator

# Initialize SageMaker session and role
sagemaker_session = sagemaker.Session()
role = "arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole"

# Create or get an Experiment
experiment = Experiment.create(
    experiment_name="house-price-prediction-exp",
    description="Tracking different ML model runs for house price prediction",
    sagemaker_boto_client=sagemaker_session.boto3_client("sagemaker"),
)

# Create a Trial within the Experiment
trial = Trial.create(
    trial_name="trial-001",
    experiment_name=experiment.experiment_name,
    sagemaker_boto_client=sagemaker_session.boto3_client("sagemaker"),
)

# Define hyperparameters
hyperparameters = {
    "eta": 0.1,               # learning rate (XGBoost uses 'eta' for learning rate)
    "max_depth": 5,
    "num_round": 100,         # number of boosting rounds
    "early_stopping_rounds": 10,
}

# Define the Estimator (XGBoost built-in example)
estimator = Estimator(
    image_uri=sagemaker.image_uris.retrieve("xgboost", sagemaker_session.boto_region_name, version="1.3-1"),
    role=role,
    instance_count=1,
    instance_type="ml.m5.large",
    output_path="s3://my-bucket/house-price-model/",
    hyperparameters=hyperparameters,
    sagemaker_session=sagemaker_session,
)

# Associate Estimator Training Job with the Trial and log hyperparameters/metrics.
# Execute estimator.fit() inside the Run context so the training job is associated automatically.
with Run(
    experiment_name=experiment.experiment_name,
    trial_name=trial.trial_name,
    sagemaker_session=sagemaker_session,
) as run:
    # Log hyperparameters explicitly (optional — some are auto-captured)
    for name, value in hyperparameters.items():
        run.log_parameter(name, value)

    # Start the training job (the job will be recorded as a trial component)
    estimator.fit("s3://my-bucket/house-price-data/")

• Always call estimator.fit(…) inside the Run context. That ensures the training job becomes a trial component and training metadata are captured automatically.
• The SDK often captures container image, algorithm name, and training-job identifiers for you; explicitly logging hyperparameters and custom metrics improves clarity and searchability.
• Use algorithm-appropriate hyperparameter names (e.g., XGBoost uses “eta” and “num_round”).

​

Where do results appear?

• SageMaker Studio Classic: exposes the original SageMaker Experiments visual dashboard with charts and comparison views.
• Newer Studio experiences: may surface MLflow or other experiment UIs; consider integrating MLflow or a third-party tool if you rely on a consistent UI.

• Visualize metrics across trials (line charts, bar charts).
• Compare multiple runs with side-by-side metric views and confusion matrices.
• Filter runs by metric thresholds or parameter values to find promising experiments quickly.

​

Summary and recommended next steps

• Experiment tracking is essential to organize model development and to make results reproducible.
• SageMaker Experiments (via the SDK) provides a hierarchical model: Experiment → Trial (use Run contexts) → Trial Component.
• Execute training and evaluation jobs inside Run contexts to automatically capture hyperparameters, metrics, and artifacts.
• Because Studio experiences are evolving, evaluate whether to use the native SageMaker Experiments UI, MLflow, or a third-party tool depending on your long-term needs for UI continuity and collaboration.

• SageMaker Experiments docs: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-experiments.html
• MLflow: https://mlflow.org/
• Weights & Biases: https://wandb.ai/
• Neptune: https://neptune.ai/
• TensorBoard: https://www.tensorflow.org/tensorboard