How do examples in a prompt change what the model does?

It is tempting to assume an example works by teaching the model the right answer. A 2022 study tested that assumption and got a surprising result. When the researchers kept the examples but scrambled their labels, pairing inputs with random wrong answers, performance barely dropped. As the paper reports, “randomly replacing labels in the demonstrations barely hurts performance” across many tasks and models.^[1]

So if the correct labels are not doing the heavy lifting, what is? The same study found that examples mainly show the model three things: “(1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence.”^[1] In plainer terms, the examples reveal the set of answers it is allowed to give, the kind of text it will be reading, and the exact shape each answer should take. The examples set the stage, and the model then performs on that stage.

Tool providers give practical advice that lines up with this. Anthropic’s guidance says, “Examples are one of the most reliable ways to steer Claude’s output format, tone, and structure,” and recommends including 3 to 5 examples for best results.^[2] OpenAI’s guide adds that when you supply examples, “try to show a diverse range of possible inputs with the desired outputs,” so the model does not latch onto an accidental pattern.^[3]