How do I know when it’s making things up?

This question almost always arrives after someone has been caught out once. A figure that turned out to be wrong, a rule that did not exist, a supplier that was not real. The embarrassment is small but it lands hard, because up to that point the thing had been so reliably helpful.

The useful thing to understand is that it was not lying to you, and it was not having an off day. It genuinely did not know.

It cannot tell either

These tools work by producing the most plausible next piece of text. A sentence that happens to be true and a sentence that merely sounds exactly like a true one are, from the inside, the same kind of thing. There is no internal flag that goes up when it crosses from one to the other.

Which means the usual human signal is missing. When a person is unsure they hedge, they slow down, their voice changes. This does not. It is equally fluent and equally confident when it is right and when it has invented something whole. Confidence tells you nothing here, and that is the entire problem in one sentence.

Where it actually goes wrong

The errors are not random. In practice they cluster, roughly in this order.

• Numbers and dates. Prices, percentages, thresholds, when something changed.
• Names. People, companies, products, job titles — especially combinations of them.
• References. Citations, section numbers, case names, links. An invented URL is extremely common and extremely convincing.
• Quotes. Who said what. It will produce a quote that sounds exactly like the person and that they never said.
• Anything small and local. A New Zealand rule, a regional supplier, your own business. The less that was written about something, the more it fills the gap itself.
• Anything recent. There is a date after which it simply was not taught anything, and it does not always volunteer that.
• Arithmetic over a list you gave it. It will often get the method right and the total wrong.

Look at that list and notice the pattern: the more specific and the more checkable a claim is, the more likely it is to be the wrong one. That is backwards from how it feels to read, which is why it catches people.

Four things that genuinely help

Give it the source instead of asking it to remember one

There is an enormous difference between “what does the contract say about termination” and “here is the contract, what does it say about termination”. Paste the document in. Then add the instruction that does the real work: answer only from what I have given you, and if it is not in there, say so.

This single change removes most of the problem for most everyday tasks, because you have stopped asking it to recall and started asking it to read.

Ask for the source, then open it

Asking where something came from is worth doing, but it does not make the answer true — a made-up claim can come with a made-up source attached, in the correct format, looking entirely respectable. The check is opening the link. If it does not resolve, or it resolves to something that does not say what was claimed, you have your answer.

Ask again in a fresh conversation

Put the same question to it again from a clean start and see whether you get the same specifics. Matching answers are not proof of anything. Answers that differ on the numbers are close to proof that it is generating rather than recalling, and that is a useful thing to know in thirty seconds.

Decide what you are actually checking

Checking everything is not realistic and would remove the point of using it. The sensible line is about consequence. A first draft of an internal email needs no checking at all. A number that goes into a quote, a statement about what the law requires, anything that ends up on your website or in front of a customer — every specific in it gets checked, every time.

Two traps worth knowing about

The first is that it is at its most persuasive precisely where it is least reliable. Vague, hedged writing is usually safe and usually useless. The crisp specific detail is the valuable part and the risky part at the same time, which is an uncomfortable arrangement but a true one.

The second is that asking “are you sure?” is not a test. These tools lean towards agreeing with you. Push back on an answer that was actually correct and it will frequently apologise and change it. So pressing it tells you about its manners, not about the facts. Go and look at the source instead.

What it is good for, then

A great deal, as long as the job is not being the final authority on a fact. It is genuinely strong at drafting, at structuring your own half-formed thinking, at summarising something you have handed it, at explaining an unfamiliar idea in plain words, and at telling you what questions you have not thought to ask yet.

Treat it as a fast, widely-read colleague with an excellent sense of how things are usually phrased and an unreliable memory for the particulars of your situation. You would not put that person's figures into a client quote without a glance. Same here, and for the same reason.