> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Testing Agents
> Explains challenges of testing LLM agents and offers strategies and best practices to validate behavior, tool usage, formats, error recovery, and integration using unit tests, mocks, evals, and E2E.
Testing agents differs fundamentally from testing a regular deterministic function. Deterministic functions always return the same output for the same input. Agents powered by large language models (LLMs) are probabilistic: given the same prompt they may call different tools, follow different reasoning paths, or produce different text across runs. Non-determinism increases test complexity, so testing must shift from asserting exact outputs to validating behaviors and properties.
## Why agent testing is hard
* Non-deterministic outputs\
LLMs usually produce different text each run unless forced toward determinism (e.g., `temperature=0`). Relying on exact-string equality is brittle.
* Multiple valid answers\
Tasks such as summarization, creative writing, or troubleshooting admit many acceptable responses; a single phrasing is rarely the only correct one.
* External dependencies\
Tools often call external APIs that can be slow, flaky, or subject to change. Tests that hit real services may be unstable.
* Complex execution paths\
An agent loop can branch: the same prompt might result in two tool calls in one run and eight in another depending on model choices.
## What to test instead
Shift testing focus from exact text to observable behaviors and properties. Six practical properties to validate:
1. Tool selection — Ensure the agent chooses the right tool for the task.\
Example: A weather request should invoke the weather tool, not the calendar tool.
2. Argument correctness — Validate that tool calls include the right argument types, required fields, and formats. Bad arguments lead to hard-to-trace failures.
3. Error recovery — If a tool fails, ensure the agent retries, uses a fallback, or informs the user rather than crashing.
4. Output format — Verify schema, JSON structure, lists, or other expected formats. Schema checks are easy to automate.
5. Guardrail compliance — Confirm the agent refuses or safely handles out-of-scope or unsafe requests.
6. Task completion — Check the agent eventually produces a useful result (not necessarily perfect).
## Four practical strategies for testing agents
Use these strategies together, choosing trade-offs between cost, speed, and stability.
### Strategy 1 — Unit tests for tools
Test each tool in isolation, without LLM involvement. These are fast, deterministic, and catch many bugs before integration.
Example unit test for a calendar tool:
```python theme={null}
# tests/test_calendar_tool.py
def test_calendar_tool():
result = check_calendar(date="2026-03-05")
assert "events" in result
assert all(
"time" in e and "title" in e
for e in result["events"]
)
```
When to use: every tool you control. Run on every commit.
### Strategy 2 — Mock tool responses
Replace real APIs with deterministic mocks to exercise the agent's reasoning loop. This lets you assert which tools were called and with what arguments without network flakiness.
Example: run an agent with mock tools and assert behavior.
```python theme={null}
# tests/test_agent_reasoning.py
mock_tools = {
"check_calendar": lambda date: [
{"time": "10:00", "title": "Team standup"}
],
"search_contacts": lambda name: {"email": "sarah@email.com"}
}
response = run_agent("Schedule lunch with Sarah", tools=mock_tools)
# Example assertions you might make:
# - Did the agent call "search_contacts" with "Sarah"?
# - Did it call "check_calendar" with an appropriate date?
# - Does the response indicate a scheduled time or suggested times?
```
When to use: CI-level tests that validate decision-making and tool sequencing.
### Strategy 3 — Evaluation-based testing (evals)
When exact text cannot be asserted, use an independent judge (a separate LLM or human raters) to score outputs against quality criteria. Evals assess “good enough” rather than exact phrasing.
When to use: scenarios where natural language quality, faithfulness, or helpfulness matters.
### Strategy 4 — End-to-end tests
Run the full agent loop in realistic environments to verify integration across components. E2E tests are slow, may cost money, and can be flaky, so run them selectively (nightly, pre-release).
When to use: critical production flows and integration verification.
## Strategy summary table
| Strategy | Purpose | When to run | Cost / Stability |
| - | - | - | - |
| Unit tests | Validate tool logic in isolation | On every commit | Low cost, high stability |
| Mock-based tests | Verify agent reasoning & tool selection | CI-level; frequent | Low cost, deterministic |
| Evals | Judge language quality and usefulness | When NLP output matters | Medium-to-high cost, stable if automated |
| End-to-end | Validate full integration | Nightly or pre-release | High cost, potentially flaky |
## Putting the strategies together
* Unit tests → verify tools you control.
* Mock-based tests → verify agent reasoning given deterministic tool outputs.
* Evals → verify output quality when phrasing can vary.
* End-to-end tests → verify the whole system for realistic scenarios.
Balance these four: use unit tests for fast coverage, mocks for agent logic, evals for quality, and selective E2E tests for critical paths.
## Practical guidance and best practices
* Start with unit tests for every tool you build. They’re fast and provide strong guarantees.
* Use mocks to assert correct tool selection and argument formation.
* Add evals for outputs where natural language quality is critical, but watch cost.
* Run end-to-end tests on a schedule to catch integration regressions.
* For deterministic debugging, temporarily set `temperature=0` to reduce variance — but don’t rely on this for production tests because it reduces coverage of the model’s natural behaviors.
* Log tool calls, response metadata, and intermediate reasoning traces to make assertions and diagnose failures.
When possible, assert on the structure and semantics of outputs (schema, tool calls, presence of required fields) rather than exact strings. This reduces brittleness while still catching regressions.
## How OpenClaw approaches testing
OpenClaw’s test framework uses [Vitest](https://vitest.dev/). Their suite typically includes:
* Unit tests for individual tools
* Schema and configuration validation tests
* Integration tests for message routing
* Session persistence tests (file I/O)
OpenClaw avoids validating LLM text via exact matches; instead, it uses evals and manual review for language quality. This follows the four-strategy approach: rigorously test the parts you control, and evaluate the parts you cannot fully determine.
Useful references:
* OpenClaw case study: [AI Agents for Beginner — OpenClaw](https://learn.kodekloud.com/user/courses/ai-agents-for-beginner-openclaw-case-study)
* Vitest: [https://vitest.dev/](https://vitest.dev/)
## Summary checklist
* Unit test every tool you own.
* Use mocks to validate agent reasoning and tool selection.
* Add evals when output quality matters.
* Run selective end-to-end tests for critical flows (scheduled runs).
* Prefer assertions on behavior, structure, and format over exact string equality.
If you follow these strategies and prioritize tests by speed, reliability, and cost, your test suite will better catch agent-specific bugs and keep your system reliable.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.