Skip to main content
Imagine a game with no scoreboard. Both teams play hard, but no one can tell who is winning. That is what a loop without a scorekeeper feels like: it can run forever, but it never truly knows whether it’s making progress or doing the right thing. The scorekeeper defines what “correct” means.
The image depicts a soccer game without a scoreboard, illustrating the concept that although both teams play hard, no one can tell who is winning. Accompanying text explains that a loop without a scorekeeper runs continuously without knowing if it's winning.
What is a scorekeeper? The scorekeeper is the mechanism that converts work into a measurable result the loop can read. It might be a test suite, a build that must pass, a checklist, or a rubric another agent uses to grade work. By returning a score each cycle, the scorekeeper tells the loop whether it is improving or regressing.
The image explains the concept of a "Scorekeeper," defining what "correct" means using three elements: Test Suite, A Build That Passes, and Checklist or Rubric.
Why the score matters On every loop iteration the system runs the scorekeeper and reads the result. A higher score means real progress; a lower score signals a regression. Without this signal the loop is blind—it cannot distinguish true progress from noise and cannot operate reliably on its own.
The image highlights the importance of a scorekeeper by illustrating that without one, a loop becomes blind and unable to differentiate real progress from noise.
Traits of a trustworthy scorekeeper A scorekeeper is only useful if it’s reliable. The four essential traits are: Objective, automatic, fast, and clear — these traits turn a plain test into a trustworthy judge.
The image outlines four traits of a good scorekeeper: objective, automatic, fast, and clear. Each trait is accompanied by a brief description and an icon.
Make the score hard to fake If the loop can pass the score by cheating—faking outputs, changing the tests, or skipping essential work—the score becomes a lie. A gamable score leads to false confidence: the loop believes it succeeded while the system remains broken.
The image describes a flawed system where cheating to pass a score leads to false success, resulting in a broken tool and unreliable outcomes.
Simple mitigations
  • Keep some checks held out (hidden) from the loop during normal iterations to prevent overfitting to visible tests. Use held-out checks as final verification.
  • Make critical checks tamper-proof—store them where the loop cannot modify them (e.g., in a separate repository or protected CI job).
  • Ensure logs and artifacts needed for verification are immutable or auditable.
The image illustrates a concept titled "Simple Tricks" about keeping some checks held out to avoid overfitting, featuring graphics of a loop and checkmarks.
Never allow the loop to weaken or edit the scorekeeper. If the system can lower the bar, it will choose the easiest path to pass rather than actually fix the work.
Protecting the scorekeeper Treat the scorekeeper as an oracle: keep its definition and implementation separate from the loop itself. Use access controls, read-only storage, and guarded CI jobs so the loop cannot tamper with tests, rubrics, or scoring logic.
The image illustrates a concept called "Simple Tricks" where a bar is lowered to allow something to pass by cheating, rather than fixing the work, with a note about respecting the oracle skill covered in Module 2.
Rule of thumb Write the scorekeeper before you start the loop. Defining the score up front makes “done” explicit and lets you verify progress objectively as you iterate.
The image provides a "Rule of Thumb" with two steps: first, "Write the scorekeeper," accompanied by a checklist icon, and second, "The goal says what 'done' looks like," illustrated by a target icon.
Write the scorekeeper first. Define the goal clearly, then use automated, objective checks to measure progress. Protect those checks from being modified by the loop.
Recap
  • The scorekeeper decides what is correct and is the heart of any reliable loop.
  • A good scorekeeper is objective, automatic, fast, and clear.
  • Make the score hard to fake: hold out checks, protect scoring logic, and prevent the loop from editing tests.
The image features a "Scorekeeper" graphic displaying a score of 72, with related attributes like "Objective," "Automatic," "Fast," "Clear," and "Hard to fake" highlighted around it.
Quick checklist
  • Write tests/rubrics before development.
  • Automate scoring in CI.
  • Keep some tests secret (held-out).
  • Store scoring logic read-only and audit changes.
  • Monitor score trends, not just single-pass results.
Further reading and references The loop can only run well when it has a trustworthy judge.

Watch Video