Try this first

Add a case for something with no exact answer:

“Summarise notes.txt in one sentence.”

Now write the check. You cannot. There is no string to compare against, because a hundred
different sentences would all be correct.

Using a model as the scorer

Ask a model whether the output meets a standard:

JUDGE = """You are grading one output against a standard. Answer with JSON only.

Standard: {standard}

Output: {output}

Reply: {{"pass": true or false, "reason": "one sentence"}}"""


def judge(output, standard):
    reply = judge_llm.send(
        [{"role": "user", "content": JUDGE.format(standard=standard, output=output)}]
    )
    return json.loads(reply.text)

Then:

Case(
    name="summarise notes",
    prompt="Summarise notes.txt in one sentence.",
    setup=git_reset,
    check=lambda: judge(
        last_answer(),
        "One sentence that mentions all three tasks in notes.txt: "
        "milk, the plumber, and the report.",
    )["pass"],
),

This works, and it opens up a large category of cases you could not otherwise score.

Where the judge lies

Four failure modes, and all four have bitten people who trusted the number:

It is generous. Asked “is this good?”, a model tends to say yes. A vague standard produces
a pass rate near 100% that means nothing.

It rewards fluency. A confident, well-written wrong answer scores better than a hesitant
right one. This is the most dangerous one, because it selects for exactly the failure mode
that is hardest to catch elsewhere.

It is inconsistent. Judge the same output twice, get different verdicts, if your standard
leaves room.

It agrees with itself. A model judging output from the same model family shares its blind
spots. If both think a subtle bug is fine, you learn nothing.

Making the judge trustworthy

Write the standard as criteria, not vibes. Not “is this a good summary?” but “does it
mention all three tasks, in one sentence, without adding anything not in the file?” Specific
criteria are checkable; adjectives are not.

Ask for a verdict per criterion. Three booleans beat one, and they tell you which part
failed.

Validate the judge. This is the step everyone skips and it is the one that matters. Take
twenty outputs, grade them yourself, then run the judge on the same twenty. If it disagrees
with you on four, your judge is wrong 20% of the time and every number it produces carries
that error.

Prefer code where code works. If the check can be a string match, an exit code, or a file
existing, use that. A model judge is for what code cannot decide, not a default.

The hierarchy

Cheapest and most reliable first:

Most agent cases live in the top row. Reach down only when you must.

A model judge is generous, likes confident prose, and can agree with its own mistakes.
Give it explicit criteria and check it against your own grading before you trust its number.

Try this before the next lesson

Add one judged case. Then grade ten outputs yourself and compare with the judge.

Count the disagreements. That count is the error bar on every judged case in your eval set,
and you should know it before you quote a number from it.