In-context learning
What is the difference between zero-shot and few-shot prompting?
Zero-shot prompting asks the model to do a task from an instruction alone. Few-shot prompting adds a few worked examples first, which usually makes the answer more accurate and consistent.
The number in “zero-shot” and “few-shot” counts how many examples you show the model before asking it to perform.
Zero-shot means no examples. You give an instruction and the question, and that is all. “Classify this review as positive or negative: ‘The food was cold.’” The model has to work out what you want from the instruction alone.
Few-shot means you include a small number of worked examples first, then your real question. You might show two or three reviews already labeled positive or negative, and only then add the unlabeled one. OpenAI’s guide describes this directly: “Few-shot learning lets you steer a large language model toward a new task by including a handful of input/output examples in the prompt, rather than fine-tuning the model.”[2] (“One-shot” is the in-between case of exactly one example.)
The GPT-3 paper put this idea to the test at scale. Its headline finding was that “scaling up language models greatly improves task-agnostic, few-shot performance,” sometimes coming close to results that previously needed fine-tuning.[1] Larger models, in other words, get more out of the examples you give them.
In practice, zero-shot is the place to start because it is the least work. If the answers come back wrong, in the wrong format, or inconsistent, adding a few good examples is often the quickest fix. The next page covers what those examples are actually doing.
References
- Language Models are Few-Shot Learners — Brown et al., 2020
- Prompt engineering guide — OpenAI