> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Release Management With Shadow Release
> Explains using shadow releases to evaluate new generative AI models on real traffic safely by routing requests to primary and shadow models and analyzing offline metrics.
In this lesson you’ll learn how to manage safe rollouts for Generative AI (GenAI) applications using a shadow release pattern. Shadow releases allow you to validate new models or prompt changes against real traffic without exposing users to unvetted behavior.
Overview: GenAI systems are non-deterministic — a model upgrade, different multi-turn prompt logic, or switching to a new model family can change outputs in ways that are difficult to predict. Shadow releases reduce risk by letting you evaluate candidate models using production traffic while returning only the primary model’s result to users.
Shadow releases let you "shadow" production requests to candidate models for offline evaluation. This enables evidence-driven decisions for promoting new models or prompt changes without impacting users.
## The problem: unpredictable behavior after release
GenAI models do not always behave like deterministic software. A change that passed tests can still produce unexpected outputs in production because:
* Model versions differ in reasoning, response style, or hallucination behavior.
* Prompt edits can alter answers in subtle ways.
* Tokenization and response length can change cost and latency characteristics.
These uncertainties create real user-experience risk after a release. Shadow release is a practical pattern to mitigate that risk.
## Shadow release pattern (high level)
How it works:
* User submits a request to your application.
* The app routes the request to the primary model and returns that response to the user.
* In parallel, the identical request is sent to one or more shadow (candidate) models.
* Shadow responses are recorded (logs, S3, analytics pipeline) and never shown to users.
* Offline analysis compares primary vs. shadow across accuracy, tone, latency, cost, and other metrics to determine promotion decisions.
Typical architecture: API Gateway → Lambda (request router) → Bedrock inference calls to primary and shadow models. The router returns the primary output immediately while storing shadow outputs for later analysis.
## Implementation notes and options
Where to implement the request router:
* AWS Lambda (simple, serverless).
* Containerized service (ECS, Fargate, EKS) for higher throughput.
* Embedded in a Java/Node.js service if you control the app runtime.
* AWS Step Functions for orchestrating parallel calls and aggregating results with visual workflows.
Design considerations:
* Shadow responses must never affect the primary response path.
* Persist shadow outputs to an analytics store (S3, CloudWatch Logs, or a dedicated pipeline) for comparison.
* Make shadow calls concurrently to avoid adding latency to the primary path (or isolate latency by not waiting for shadow calls in the user-facing response).
* Track token usage and latency per model to estimate cost and performance impact at scale.
Shadow responses must never be surfaced to users or change the user-facing result. Ensure logging and persistence are isolated from the primary response path.
## Example: Lambda request router (Python)
This example demonstrates the end-to-end flow in a Lambda handler:
* Read incoming request
* Call the primary model and return its result
* Call the shadow model(s) with the same prompt
* Log or persist shadow outputs for offline analysis
Adjust model IDs, retry/backoff logic, and logging sinks for your environment.
```python theme={null}
import json
import boto3
import concurrent.futures
bedrock = boto3.client("bedrock-runtime")
PRIMARY_MODEL = "amazon.nova-lite-v1:0"
SHADOW_MODELS = ["amazon.nova-pro-v1:0"] # add more candidates as needed
def call_model(model_id, messages):
"""Invoke Bedrock and return the plain text response and metadata."""
response = bedrock.converse(modelId=model_id, messages=messages)
text = response["output"]["message"]["content"][0]["text"]
# Optionally capture tokens, latency, and raw response for analysis
return {"model": model_id, "text": text, "raw": response}
def lambda_handler(event, context):
body = json.loads(event.get("body", "{}"))
user_input = body.get("message", "")
messages = [{"role": "user", "content": [{"text": user_input}]}]
# Call primary model synchronously (returned to user)
primary_result = call_model(PRIMARY_MODEL, messages)
primary_text = primary_result["text"]
# Call shadow models concurrently, do not await for response to return to user
with concurrent.futures.ThreadPoolExecutor(max_workers=len(SHADOW_MODELS)) as executor:
futures = {
executor.submit(call_model, model, messages): model for model in SHADOW_MODELS
}
for fut in concurrent.futures.as_completed(futures):
try:
shadow_result = fut.result()
# Persist the shadow result for offline analysis (CloudWatch, S3, DB, etc.)
print("SHADOW:", shadow_result["model"], shadow_result["text"])
# Optionally store `shadow_result["raw"]` for metric extraction
except Exception as e:
# Handle and log individual shadow call errors without impacting the primary response
print("Shadow call error:", e)
# Return the primary model's response to the user
return {"statusCode": 200, "body": json.dumps({"response": primary_text})}
```
## Metrics to collect and compare
Collect multiple dimensions so promotion decisions are evidence-driven:
| Metric | Why it matters | Example storage |
| - | - | - |
| Accuracy / correctness | Ensures the candidate produces correct task outputs | `S3` or analytics DB |
| Tone & alignment | Aligns with product voice, guardrails, safety | `CloudWatch Logs`, review tool |
| Latency | Affects UX and concurrency needs | `CloudWatch Metrics` |
| Cost (tokens) | Different tokenization => different cost at scale | `S3` + token counters |
| Failure/retry rates | Model or network instability | `CloudWatch` |
Tokenization differences can materially affect cost when operating at scale — measure tokens per-request per-model.
## When to use a shadow release
Use shadow releases for scenarios where behavior may change and you want low-risk evaluation:
* Model upgrades (new foundation model versions)
* Prompt or system-message changes
* New product features (agents, decision logic, guardrails)
* Performance or cost optimization experiments
You can also use shadow releases alongside other strategies:
* Canary releases: expose a small percentage of live traffic to a candidate model.
* A/B testing: return different content to different users for direct comparison (user-visible).
Shadow release differs because it uses real traffic while keeping user experience unchanged.
## Expected outcomes and benefits
* Safer production releases: observe candidate behavior on real requests before promotion.
* Evidence-driven promotion: promote models based on measured metrics rather than intuition.
* Continuous improvement: iterate on prompts, guardrails, and model selection with minimal user risk.
* Operational visibility: capture behavior changes (latency, token usage, hallucinations) early.
## Key takeaways
* GenAI systems require careful release strategies because model/version changes can produce very different results.
* Shadow releases let you evaluate real-world changes safely without exposing users to unvetted behavior.
* Combine metrics (accuracy, tone, latency, and cost) and automated pipelines to make promotion decisions reliable and repeatable.
This concludes the short lesson on release management with shadow releases. The course also includes an introduction to Bedrock AgentCore and orchestration techniques for more advanced release workflows.
## Links and references
* Amazon Bedrock documentation: [https://docs.aws.amazon.com/bedrock/](https://docs.aws.amazon.com/bedrock/)
* AWS Lambda: [https://docs.aws.amazon.com/lambda/](https://docs.aws.amazon.com/lambda/)
* AWS Step Functions: [https://docs.aws.amazon.com/step-functions/](https://docs.aws.amazon.com/step-functions/)
* Best practices for canary and A/B testing: [https://aws.amazon.com/what-is/continuous-delivery/](https://aws.amazon.com/what-is/continuous-delivery/)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.