Reasoning & chain of thought
Can a model reason step by step without any examples?
Yes. Just adding the phrase 'Let's think step by step' prompts a model to reason out loud before answering, with no examples in the prompt. This is zero-shot chain of thought.
The original chain-of-thought method needs you to write out worked examples in the prompt. A 2022 paper by Takeshi Kojima and colleagues found a much simpler trigger: just add the words “Let’s think step by step” before the model answers, with no examples at all.[1]
That one phrase had a striking effect. On a set of arithmetic word problems called MultiArith, accuracy went from 17.7 percent to 78.7 percent. On the harder GSM8K math set it rose from 10.4 percent to 40.7 percent. Both numbers are for the same model; the only change was the added phrase.[1]
This is called zero-shot chain of thought, “zero-shot” meaning no examples were provided. It is worth knowing because it is almost free to try. If a model gives a wrong answer to a multi-step question, asking it to think step by step before answering is often the quickest thing to attempt, and it costs only a few extra words in the prompt.
The catch is that the model still produces a longer answer, which takes more time and more tokens. The next pages cover models that do this kind of reasoning by default, and how to judge when the extra steps are worth it.
References
- Large Language Models are Zero-Shot Reasoners — Kojima et al., NeurIPS 2022