Capabilities & limits
Does AI really understand, and why is it bad at counting?
A model arranges language by statistical pattern, not by grasping meaning. The same pattern-matching is why it fumbles exact tasks like counting letters or doing arithmetic.
It is easy to mistake fluent writing for understanding. A useful corrective is the phrase stochastic parrot, from a 2021 paper, which describes a model as something that statistically mimics text without real understanding.[1] The paper puts it sharply: a model is “stitching together sequences of linguistic forms … according to probabilistic information about how they combine, but without any reference to meaning.”[1] The text comes out coherent because the patterns in the training data were coherent, not because the model knows what it is saying.
This is not a claim that models are useless. It is a claim about how they work, and it predicts where they break. The clearest example is exact computation. Models are pattern learners, not step-by-step calculators, so any task that needs a precise procedure rather than a likely-looking answer is a weak spot.
The famous case is counting letters: leading models often miscount the number of “r” letters in “strawberry,” a task a child can do.[2] A study of this found that models can recognize the letters in a word but fail to count them, and the errors track the complexity of the counting operation, with most models unable to count correctly when a letter appears more than twice.[2] The same shortfall shows up in arithmetic with large numbers and other tasks that reward exactness over plausibility.
The fix is not to argue the model into understanding. It is to hand exact work to something exact: let the model call a calculator, run code, or use a tool, and keep it doing what it is genuinely good at, which is the language around the answer.
References
- Stochastic parrot — Wikipedia
- Why Do Large Language Models (LLMs) Struggle to Count Letters? — arXiv (Fu, Ferrando, Conde, Arriaga, Reviriego)