The thing everyone hits in week one

You ask the model to use some library. It writes code that calls
library.saveToFile(data). The code looks perfect. It has the right style, the right
naming, sensible arguments.

You run it. There is no saveToFile function. There never was.

People call this a hallucination: the model states something that is not true, in the
same confident tone it uses for true things.

It is not a malfunction

Here is the part that surprises people. Nothing broke. The model did exactly what it always
does.

It was asked to continue text. In all the code it has read, a library like that one usually
has a function like that one, with a name like that one. So saveToFile is a very likely
next piece of text. The model produced likely text. That is its only job.

The problem is not that the model lied. The problem is that it has no way to tell the
difference between “this exists” and “this is the kind of thing that usually exists”.

Why it sounds so sure

There is no small voice inside the model saying “I am only sixty percent sure about this”.
It does not keep a list of what it knows and what it does not.

The confident tone is not a signal of correctness. It is a writing style it learned from
documentation and textbooks, which are also written confidently. It uses that same style
whether it is right or wrong.

This is worth saying plainly, because it is the single most expensive mistake beginners
make:

How sure the model sounds tells you nothing about whether it is right.

Module 5 is built on this one sentence.

The good news

Hallucinated code usually fails loudly. You run it, and it breaks. That is annoying but
safe.

The dangerous kind is code that runs perfectly and does the wrong thing quietly. You will
meet a real example of that in Module 4, when we build the review scheduler for our app.

Try this before the next lesson

  1. Ask a model for the functions available in a library you know well. Check every one.
  2. Ask it about a library you invent on the spot, for example “the chaiwala npm package”.
    See whether it tells you it does not know, or invents an answer.
  3. Ask it the same factual question twice in two separate chats. Do the answers match?
PreviousNext