Choosing a model
How should you start choosing a model?
Do not start by ranking models. Start by writing down the task and the criteria that matter for it. Anthropic's own guidance is that knowing these answers in advance makes narrowing the choice much easier.
The mistake is to start with the leaderboards. A better order is to start with your task: what are you actually asking the model to do, and how would you know it did it well? The model comes last, after you know what you need.
Anthropic’s own guidance frames model choice as balancing a few criteria for your specific case, and it is blunt about the value of doing this first: “Knowing these answers in advance will make narrowing down and deciding which model to use much easier.”[1]
So write the criteria down before you compare anything. What capabilities does the job require? How fast must the answer come back? What is your budget? Where is the data allowed to go? A task with no privacy constraint and loose latency has very different answers than one handling medical records in real time. The next pages take these one at a time.
References
- Choosing the right model — Anthropic