Try this first

Run one task through Rover five times, from the same starting state:

for i in 1 2 3 4 5; do
  git checkout . && python3 agent.py "Add a word_count tool to agent.py"
done

Five runs. Different numbers of turns, different edit points, and probably a different outcome
on at least one.

Now answer this: did your last change make Rover better?

You cannot say. You have no baseline, and one run tells you nothing.

Why normal testing does not fit

A unit test asserts an exact output. An agent does not have one.

There are three different sources of variation, and they need different responses:

The model is not deterministic. The same prompt gives different text.

The path is not fixed. It may read three files or five, in any order, and both can be
correct.

“Correct” is a range. A good summary is not one string. Two different implementations of
word_count can both be right.

Asserting on exact output fails all three. But the answer is not “you cannot test agents”. It
is that you assert on something else.

Assert on outcomes, not paths

Go back to Lesson 4.2, where you wrote checks:

("word_count is defined", lambda: "def word_count" in read("agent.py")),
("agent.py still imports", lambda: run("python3 -c 'import agent'") == 0),

Those work on all five runs. They do not care how many turns it took or which order the files
were read. They ask: is the world in the right state now?

That is the shape of an agent test. Not “did it say the right thing” but “did the right thing
happen”.

The three things worth measuring

The third is the one people skip, and for agents it is often the most useful. An agent that
succeeds nine times in ten is a different product from one that succeeds five times in ten,
and a single run cannot tell them apart.

Run it more than once

The consequence of everything above:

A single run is not a result. It is one sample from a distribution.

Five runs is enough to catch obvious regressions. Ten gives you a number you can compare
between versions. For a change you think is small, ten runs before and ten after is the
minimum honest comparison.

That sounds expensive until you price it against shipping a change that made your agent worse
and not finding out for two weeks.

An agent gives a different answer every time, so check outcomes rather than output, and run
it enough times to see a rate rather than an anecdote.

Try this before the next lesson

Take your five runs from the top of this lesson. For each, record: did it work, how many
turns, how many tokens.

You now have your first data. It probably shows more variation than you expected — and that
variation is what a single run was hiding from you all course.