When did deep learning take off?

The ideas from the early years finally met the data and the hardware they needed. In under a decade, neural networks went from a fringe technique to the default way to do machine learning.

• 2009 — ImageNet. Fei-Fei Li and colleagues first presented ImageNet, a very large labeled image database, at the CVPR conference. The annual ImageNet recognition challenge that grew out of it began in 2010 and became the proving ground for the next milestone.^[1]

• 2012 — AlexNet. A deep neural network built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton won the ImageNet challenge with a top-5 error rate of 15.3%, far ahead of the runner-up. The entry was submitted on 30 September 2012, and its win convinced the field that deep learning trained on GPUs was the way forward.^[2]

• 2016 — AlphaGo. DeepMind’s AlphaGo beat the professional Go champion Lee Sedol four games to one in a match played in Seoul from 9 to 15 March 2016. Go had long been considered too complex for computers, so the result drew wide attention.^[3]

• 2017 — The transformer. Researchers at Google published “Attention Is All You Need” on 12 June 2017, introducing the transformer architecture built around the attention mechanism. Nearly every large language model since is a descendant of this design.^[4]