Skip to main content
In this demo we’ll add a second variant of an existing XGBoost model and configure a SageMaker endpoint to perform A/B testing (traffic splitting) and/or shadow testing. This walkthrough shows how to deploy a second model variant using SageMaker JumpStart, create or update an endpoint configuration, enable data capture, and use Model Monitor to analyze captured data.
  1. Deploy a second model variant with SageMaker JumpStart
  • Open SageMaker JumpStart and choose the XGBoost classification solution.
  • Deploy a new model, selecting an appropriate instance type such as ml.m5.2xlarge.
  • Complete the deployment process and wait for the model to be created.
The image shows a screenshot of AWS SageMaker Studio's deployment interface, specifically the "Deploy model to endpoint" section where an instance type is being selected.
  1. Confirm models in the SageMaker console
  • Navigate to SageMaker Console → Dashboard → Models to see all model artifacts.
  • You should see the newly created model alongside your previously deployed model(s).
The image shows the Amazon SageMaker console with a list of machine learning models, including xgboost-classification and xgboost-regression, along with their creation times.
  1. Create or update an endpoint configuration
  • Select the model you want to expose and click Create endpoint.
  • If you don’t have an existing endpoint configuration, create a new one with a descriptive name.
  • Choose a provisioned endpoint (recommended for real-time inference).
  • Configure encryption and data capture as needed. Enabling data capture here will automatically record request/response payloads for Model Monitor to consume later.
When configuring endpoint variants you’ll see entries labeled Production (often shown as “P”) and Shadow (often shown as “S”). Production variants serve live traffic and return responses to clients. Shadow variants receive mirrored requests for testing; their responses are not returned to the caller.
The image displays an Amazon SageMaker console screen showing the configuration of an endpoint, with a model named "xgboost-classification-102920" under the Production variant.
Summary of variant types and when to use them:
  1. Configure traffic splitting (A/B testing)
  • Add both models as production variants in the endpoint configuration and assign traffic weights.
  • Example for a 50/50 split:
  • Alternatively, add one model as a production variant and the other as a shadow variant if you only want to mirror traffic for testing.
Data capture is set on the endpoint configuration (you choose sampling percentage and S3 destination). Model Monitor uses the captured requests/responses for drift detection, distribution checks, and other analyses.
The image shows an Amazon SageMaker console dashboard displaying the details of an XGBoost classification model, including endpoints and a monitor schedule section.
  1. Configure data capture (sampling and destination)
  • When creating the endpoint configuration, enable data capture and set the sampling percentage to control what fraction of requests/responses are stored.
  • Captured data is what Model Monitor consumes to detect data drift, feature distribution changes, and other issues.
The image shows an Amazon SageMaker interface for setting up a data capture configuration. It includes options for entering an endpoint configuration name, selecting data capture for prediction requests/responses, and setting a sampling percentage.
  1. Evaluate and iterate using captured data and Model Monitor
  • Compare latency, input/output distributions, error rates, and other telemetry between variants using the captured data.
  • To compare accuracy (or other label-dependent metrics), you need ground-truth labels — for example, delayed feedback or a labeled validation dataset.
  • Use Model Monitor to generate reports and schedules that continuously evaluate production data and trigger alerts when anomalies or drift are detected.
  • Based on these analyses, decide to promote a candidate variant to full production or iterate on the model.
Use production variants with traffic weights for controlled live A/B tests and shadow variants to mirror traffic for safe offline evaluation. Always enable data capture if you plan to use Model Monitor — captured payloads are the foundation for drift detection, distribution comparisons, and model promotion decisions.
Links and references

Watch Video