What do layers do, and what does depth mean?

A language model does not process text in one step. It passes the text through a stack of layers, one after another. Each layer takes the running internal representation of the text, reworks it, and hands the result to the next layer. The parameters live inside these layers, so the layers are where the actual work happens.

The number of layers in the stack is the model’s depth. Bigger models tend to be deeper. For example, the LLaMA 2 family at 7B, 13B, and 70B parameters “consist of 32, 40 and 80 transformer layers.”^[1] So the 70B model is not just wider, it also runs the text through more than twice as many processing steps as the 7B.

Why stack so many? The rough intuition is that early layers tend to handle surface details and later layers build toward more abstract meaning, with each layer working from the output of the one before it. More layers give the model more chances to refine its understanding before it produces an answer. Depth is one of the main knobs a model’s designers turn, alongside how wide each layer is.