Lab Walkthrough Add Monitoring

Your safe agent currently enforces iteration limits and catches tool errors, but it lacks visibility into what happened during a run: which tools were called, how long they took, and how many tokens the model consumed. In this lesson we add simple, low-overhead instrumentation so you can debug, profile, and optimize agent executions. Overview

Environment: Python, virtualenv, and the OpenAI SDK are preinstalled.
Working directory: /root/code.
Goal: start from the safe agent and add logging for tool calls, timing, token usage, and iteration warnings.

Why instrument an agent?

Faster debugging of tool failures.
Identify slow tools and optimize or cache them.
Track token usage to control costs.
Get advance warning before hitting the iteration cap.

Baseline (what you start from)

The baseline already enforces a max iteration cap of 10 and includes try/except handling for tool execution.
Example baseline snippet (trimmed for clarity):

from openai import OpenAI
import os
import json
import time

client = OpenAI()
model = "openai/gpt-4.1-mini"
MAX_ITERATIONS = 10

def execute_tool(name, args):
    try:
        if name == "check_calendar":
            result = check_calendar(**args)
        # other tool branches...
    except Exception as e:
        result = f"Error: {str(e)}"
    return result

What we’ll add

run_log: a list of dictionaries capturing each tool call: tool name, args, truncated result, and duration_ms.
Timing inside execute_tool: measure start/end, compute duration in milliseconds, and append a compact trace to run_log.
Token tracking: maintain running totals of prompt and completion tokens after each API call.
Iteration tracking: maintain iteration_count and print a warning when within the last two iterations.
Execution summary: when the agent finishes, print iterations used, total tokens, number of tool calls, each call’s timing, and total wall-clock time.

Quick links and references

Instrumentation: add run_log and an instrumented execute_tool

Initialize run_log and token counters immediately before your agent loop.
Replace the old execute_tool with the instrumented version below. It measures duration and stores a truncated string result so logs remain compact.

def execute_tool(name, args, run_log):
    start_time = time.time()
    try:
        if name == "check_calendar":
            result = check_calendar(**args)
        elif name == "some_other_tool":
            result = some_other_tool(**args)
        else:
            result = f"Unknown tool: {name}"
    except Exception as e:
        result = f"Error: {str(e)}"
    duration = time.time() - start_time
    # Ensure we store a string (truncated) so logs stay compact
    run_log.append({
        "tool": name,
        "args": args,
        "result": str(result)[:100],
        "duration_ms": round(duration * 1000)
    })
    return result

Track tokens after each API call

Maintain running counters for prompt and completion tokens and update them after each client.chat.completions.create call.
Different SDK versions may return usage either as an attribute or a dict — handle both forms.

total_prompt_tokens = 0
total_completion_tokens = 0

# Example call to the Chat Completions endpoint using the OpenAI client
response = client.chat.completions.create(model=model, messages=messages, tools=tools)

# Update token counters if usage is present (handle both attribute and dict forms)
usage = None
if getattr(response, "usage", None) is not None:
    usage = response.usage
elif isinstance(response, dict) and response.get("usage") is not None:
    usage = response.get("usage")

if usage is not None:
    # prompt_tokens may be an attribute or a key depending on the SDK
    if isinstance(usage, dict):
        prompt_tokens = usage.get("prompt_tokens")
        completion_tokens = usage.get("completion_tokens")
    else:
        prompt_tokens = getattr(usage, "prompt_tokens", None)
        completion_tokens = getattr(usage, "completion_tokens", None)

    if prompt_tokens is not None:
        total_prompt_tokens += int(prompt_tokens)
    if completion_tokens is not None:
        total_completion_tokens += int(completion_tokens)

Iteration tracking and advance warning

Keep iteration_count and increment it at the start of each loop iteration.
Print an advance warning when you are within the last two iterations so you can take corrective action or gracefully exit.

iteration_count = 0

for iteration in range(MAX_ITERATIONS):
    iteration_count += 1
    # Warn when within the last two iterations (e.g., for MAX=10, warn on 9 and 10)
    if iteration_count >= MAX_ITERATIONS - 1:
        print(f"WARNING: approaching iteration limit ({iteration_count}/{MAX_ITERATIONS})")

    # Call the model (example)
    response = client.chat.completions.create(model=model, messages=messages, tools=tools)

    # (Handle response, update tokens, call tools via execute_tool, etc.)

Record wall-clock time and print an execution summary

Record a wall-clock start time before the loop and compute elapsed time after it completes.
Print a concise summary with iterations used, tokens, tool call counts and timings, and total elapsed time.

wall_start = time.time()

# ... agent loop runs here ...
elapsed = round(time.time() - wall_start, 2)
print("\n=== Execution Summary ===")
print(f"Iterations used: {iteration_count}/{MAX_ITERATIONS}")
print(f"Total tokens: {total_prompt_tokens + total_completion_tokens}")
print(f" - Prompt tokens: {total_prompt_tokens}")
print(f" - Completion tokens: {total_completion_tokens}")
print(f"Tools called: {len(run_log)}")
for entry in run_log:
    print(f" - {entry['tool']}: {entry['duration_ms']} ms, args={entry['args']}, result={entry['result']}")
print(f"Total elapsed time (s): {elapsed}")

Instrumentation summary table

Metric	What it records	Example
`run_log`	One entry per tool call: `tool`, `args`, truncated `result`, `duration_ms`	`{"tool":"check_calendar","args":{...},"result":"OK","duration_ms":120}`
Tokens	Aggregated `prompt` and `completion` tokens from `response.usage`	`prompt_tokens: 45`, `completion_tokens: 12`
Iterations	`iteration_count` and warnings when close to `MAX_ITERATIONS`	`WARNING: approaching iteration limit (9/10)`
Wall-clock	Total elapsed seconds for the whole run	`Total elapsed time (s): 3.42`

Full integration notes

Initialize run_log, total_prompt_tokens, total_completion_tokens, and iteration_count immediately before entering your agent loop.
Replace your existing execute_tool with the instrumented version and call it as execute_tool(name, args, run_log).
After every call to client.chat.completions.create, update token counters from response.usage.
Preserve try/except handling inside execute_tool so tool failures continue to produce descriptive results that are logged.
Keep the safety cap MAX_ITERATIONS unchanged — instrumentation should not alter agent control flow.

This instrumentation records what happened (tools called, durations, token usage, and iteration counts) without changing agent decision logic or safety limits. Use these logs to pinpoint slow tools, unexpected exceptions, or high token usage.

Security and privacy note

Be careful what you log. Logs may contain sensitive information from tool arguments or results. Mask or redact secrets before writing logs to persistent storage or sharing them.

Run it

Run the monitored agent. It will:
- Call tools (for example, check_calendar) via the instrumented execute_tool.
- Append compact entries to run_log for each tool call.
- Print a one-line answer (your agent’s response) followed by the Execution Summary block showing iterations, tokens, per-tool durations, and total elapsed time.

Example output snapshot (illustrative)

Agent answer: "I found a free slot on Tuesday at 3pm."
=== Execution Summary ===
Iterations used: 3/10
Total tokens: 120
 - Prompt tokens: 75
 - Completion tokens: 45
Tools called: 2
 - check_calendar: 150 ms, args={'date':'2026-10-07'}, result=OK
 - send_email: 320 ms, args={'to':'user@example.com'}, result=Email sent
Total elapsed time (s): 1.45

Next steps

Persist run_log to a JSON file for later analysis or attach it to a debugging UI.
Add contextual IDs to log entries (request_id, session_id) to correlate logs from multiple runs.
If costs are a concern, use the token counters to create alerts when usage exceeds thresholds.

By adding these lightweight instrumentation primitives you can quickly surface where time and tokens are spent and iterate on performance or correctness with much more confidence.

LLM Fundamentals

Tools and Workflows

Building AI Agents

Introduction

Production & OpenClaw

Lab Walkthrough Add Monitoring

Watch Video