Try this first

Ask Rover to do three things in one message:

“Add a word_count tool to agent.py, add a test for it in test_agent.py, and update
README.md to mention it.”

Then check all three files.

On a good number of runs, two are done and one is not, and Rover’s final message says all
three are complete.

What you just saw

Go back to Lesson 1.5. A reply with no tool calls means the model stopped talking.

That is all it means. It does not mean the task is complete, the files are correct, or the
claim in the final message is true. Those are facts about the world, and the reply is a fact
about the conversation.

This is the single most expensive misunderstanding in agent work, because the failure is
confident. A crash tells you it failed. This tells you it succeeded.

Why it happens

Not because the model is careless. Three ordinary mechanisms:

It lost track. Three tasks, twelve tool calls, a long transcript. Item two scrolled out of
the model’s attention somewhere around turn eight.

It thinks it did. It wrote the edit, the edit did not apply cleanly, and it did not
re-read the file. Its belief about the file is stale, and its report is honest — about a file
that does not exist.

The task was ambiguous. “Update README.md to mention it” is done by adding one line. It
added one line. You expected a section.

None of those are lies. All of them produce a false completion claim.

The fix is not a better prompt

You can improve this with prompting. “Verify each step before reporting completion” helps.
It does not solve it, because you are asking the thing that is mistaken to check whether it is
mistaken.

The fix is that you check. In code. Outside the model.

def verify(checks):
    """checks: list of (description, callable returning bool)."""
    failures = [desc for desc, check in checks if not check()]
    if not failures:
        return None
    return "These are not done yet:\n" + "\n".join(f"- {f}" for f in failures)

And in the loop, when the model wants to stop:

if not reply.wants_tool:
    problem = verify(CHECKS)
    if problem is None:
        break
    messages.append({"role": "user", "content": problem})
    continue          # send it back with the real state of the world

Now “I have finished” is a proposal, which your code accepts or rejects.

What a check looks like

Cheap and concrete. The point is that they run outside the model:

CHECKS = [
    ("word_count is defined in agent.py", lambda: "def word_count" in read("agent.py")),
    ("word_count is in the TOOLS list", lambda: '"word_count"' in read("agent.py")),
    ("there is a test for it", lambda: "word_count" in read("test_agent.py")),
    ("agent.py still imports", lambda: run("python3 -c 'import agent'") == 0),
]

The last one is the best kind: it does not check whether the model did what it said, it checks
whether the result actually works.

The general principle

This is the same idea as tests, and it arrives for the same reason. You do not trust a
confident report about code. You run something.

For an agent it matters more, because an agent produces confident reports as its normal
output, in fluent English, with no signal distinguishing an accurate one from a wrong one.

A reply with no tool calls is a proposal to stop, not proof of completion. Something outside
the model has to decide whether the work is done.

Try this before the next lesson

Write three checks for the three-part task above, and wire them in.

Then run the task five times and record how often the verify step fires. That number is the
most useful thing you will measure in this module — and in Module 7 you will make it a
permanent one.

PreviousNext