Try this first

Run any two-turn conversation and print two fields:

print(reply.raw.usage)        # your provider's own usage object

Both are zero. You have never used caching, and you have been resending the same content at
full price the whole course.

What caching is

The API can remember the beginning of your prompt and skip reprocessing it. A cache read costs
about a tenth of a normal input token. A cache write costs about a quarter more than normal.

So it pays for itself on the second request and is close to free after that. For an agent —
where the same system prompt, the same tools, and a growing shared history go out every turn —
it is the single biggest cost lever you have.

How you switch it on depends on your provider, and this is one of the few places in the
course where that matters:

Provider How caching works
OpenAI Automatic. Repeated prefixes are cached for you, with no parameter
Anthropic Explicit. You mark what to cache with a cache_control parameter
Gemini Explicit, with its own separate caching API
Ollama / local The model is already on your machine; there is no network cost to save

Anthropic’s explicit form looks like this, and it is the clearest illustration of the idea:

# Anthropic-specific — check your provider's own documentation.
response = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    cache_control={"type": "ephemeral"},
    system=SYSTEM_PROMPT,
    tools=TOOLS,
    messages=messages,
)

Whether you switch it on or your provider does it silently, the rule in the next section
decides whether it works at all — and that rule is the same everywhere.

The one rule

Everything else in this lesson follows from a single fact:

Caching is a prefix match. The API caches from the start of your prompt up to a marked
point. If any byte in that prefix changes, the cache is invalid from that byte onwards.

The prompt is assembled in a fixed order:

tools  →  system  →  messages

Tools first. So a change to your tool list invalidates everything. System prompt next; a
change there invalidates all the messages. Messages last, and each new turn extends the
prefix, which is why a growing conversation caches beautifully — the old part never changes.

How people break it

Here is a system prompt that costs you every cache hit for the entire session:

SYSTEM_PROMPT = f"""You are Rover, a coding assistant.
The current date is {datetime.now()}.
You are helping {user.name} in {os.getcwd()}.
"""

Three invalidators in four lines. The timestamp changes every call, so the prefix is different
every call, so nothing ever caches. There is no error. Your bill is just three times what it
should be, and cache_read_input_tokens stays at zero.

The usual suspects:

Pattern Why it breaks caching
datetime.now() in the system prompt New prefix every request
A UUID or request ID near the top Same
json.dumps(d) without sort_keys=True Key order can vary between runs
Tools built per user or per session Tools render first — nothing after them caches
Adding or removing a tool mid-session Invalidates the whole prefix

The fix is ordering, not markers

Put stable content first and volatile content last. That is the whole technique.

If Rover needs to know the date, it does not go in the system prompt. It goes in the message,
at the end, where it invalidates nothing:

messages.append({
    "role": "user",
    "content": f"[today is {date.today()}]\n\n{user_message}",
})

Same information. Same behaviour. Full cache hits.

Checking your work

There is no way to tell by reading. Measure — and the field names are provider-specific, so
look yours up once:

Provider Where the cache hit shows up
Anthropic usage.cache_read_input_tokens and usage.cache_creation_input_tokens
OpenAI usage.prompt_tokens_details.cached_tokens
Gemini usage_metadata.cached_content_token_count

reply.raw gives you the untouched response, which is where these live:

reply = llm.send(messages, TOOLS)
print(reply.raw.usage)          # look once, find your provider's field, then track it

By turn three of any session, most of your input should be cache reads. If it is zero, you
have an invalidator, and it is nearly always in the first thousand tokens.

One trap that catches people on every provider: the plain input-token count is not your
prompt size once caching is on. It is only the uncached remainder. Add the cached figures in
too, or you will badly under-read your own usage and think a long session is cheaper than it
is.

Caching is a prefix match. Stable content first, volatile content last, and check
cache_read_input_tokens — a broken cache is silent.

Try this before the next lesson

Add cache_control and print the cache fields every turn.

Then deliberately put datetime.now() at the top of your system prompt and watch the reads
drop to zero. Take it out and watch them come back. Ten seconds of work, and you will never
mis-diagnose this again.

PreviousNext