In-context learning
What is in-context learning?
In-context learning is the ability of a large language model to pick up a new task from what is written in the prompt, at the moment it answers, with no retraining and no change to its weights.
When you give a language model a fresh task in the prompt and it just does it, something quietly surprising is happening. The model is not being retrained. Its internal weights, fixed when training ended, do not move. It is picking up the task from the words in front of it and applying that pattern to your question. This is called in-context learning.
The idea came into focus with the 2020 paper that introduced GPT-3, a model with 175 billion parameters. Its central claim was that a large enough model can handle new tasks from a prompt alone. As the authors put it, the model is “applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction.”[1] “Gradient updates” is the technical name for the training step that adjusts a model’s weights, so saying there are none means the model is not learning in the usual sense.
It helps to hold two kinds of learning apart. Ordinary training changes the weights, slowly, over huge amounts of data, and the result is baked in. In-context learning changes nothing permanent. It is more like reading the instructions on a form and filling it out correctly the first time. The “learning” lasts only for that one prompt, and the next conversation starts fresh.
This property is why prompting matters so much. Because the model adapts to whatever you put in the context, the way you write the prompt, and especially the examples you include, becomes the main lever you have over its behavior.
References
- Language Models are Few-Shot Learners — Brown et al., 2020