Revival

Rise of Statistical Methods

In the 1990s, statistical methods began to dominate AI research. Hidden Markov Models (HMMs) in speech recognition, Naive Bayes and decision trees in machine learning, all achieved good results.

Birth of the World Wide Web

Tim Berners-Lee created the World Wide Web. The internet’s proliferation changed human lifestyle and provided unprecedented data resources for AI.

Support Vector Machine

Vladimir Vapnik et al. proposed Support Vector Machine (SVM), a powerful classification and regression method. SVM achieved excellent performance in text classification and handwritten recognition.

Vladimir Vapnik

Long Short-Term Memory (LSTM)

Sepp Hochreiter and Jürgen Schmidhuber proposed LSTM, a special type of RNN capable of learning long-term dependencies. LSTM achieved great success in speech recognition and language modeling.

Sepp Hochreiter & Jürgen Schmidhuber

LeNet-5

Yann LeCun developed LeNet-5, a CNN for handwritten digit recognition. LeNet-5 was among the first deep learning models deployed at scale commercially — used by AT&T/NCR for check reading starting in 1993 and by the US Postal Service for ZIP code recognition from the late 1990s.

Yann LeCun

Neural Probabilistic Language Model

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin proposed a neural probabilistic language model that jointly learns distributed word representations and word-sequence probabilities to address the curse of dimensionality. The model is feed-forward, not recurrent.

Yoshua Bengio

CIFAR Neural Computation Program

Beginning in 2004, a CIFAR program brought Geoffrey Hinton, Yann LeCun, Yoshua Bengio, and collaborators into regular exchange. It helped coordinate research that contributed to the deep-learning revival; it was not a single universal starting point.

Geoffrey Hinton

Deep Belief Networks

Geoffrey Hinton and his colleagues proposed Deep Belief Networks (DBN), with a fast, greedy algorithm that trains them one layer at a time. This layer-wise pre-training helped revive interest in deep neural networks.

Geoffrey Hinton

GPU for Deep Learning

Researchers began using GPUs to accelerate deep learning training. GPU’s parallel computing made training large neural networks possible.

ImageNet Dataset

The ImageNet team published its large-scale hierarchical image database paper in 2009, organizing labeled images with the WordNet hierarchy. ImageNet and its later recognition challenge became important infrastructure for large-scale computer-vision training and evaluation.

Fei-Fei Li

Speech Recognition Breakthrough

Microsoft Research and the University of Toronto pioneered the first major deep-learning breakthrough in speech recognition (Dahl, Yu, Deng 2011; Mohamed, Dahl, Hinton 2012), achieving significant word-error-rate reductions. Google deployed DNN/LSTM-based speech recognition in production starting in 2012–2015.

ImageNet Breakthrough

Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton’s team entered AlexNet in the 2012 ImageNet competition (ILSVRC) and won by a landslide: a top-5 error of 15.3%, against 26.2% for the runner-up.

AlexNet Team

Word2Vec

Tomas Mikolov released Word2Vec, a neural network model for learning word embeddings. Word2Vec enabled computers to understand semantic relationships between words.

Tomas Mikolov

Generative Adversarial Networks (GAN)

Ian Goodfellow proposed GAN, one of the most important innovations in deep learning. GAN generates extremely realistic images through adversarial training.

Ian Goodfellow

ResNet

Kaiming He et al. proposed ResNet, solving the training difficulty of deep networks through skip connections. ResNet enabled training hundreds of layers.

Kaiming He

AlphaGo Defeats Lee Sedol

DeepMind’s AlphaGo defeated world Go champion Lee Sedol 4-1. Go was considered one of the most difficult board games for AI.

DeepMind AlphaGo

Transformer Architecture

Google proposed the Transformer architecture, a revolutionary breakthrough in NLP. Transformer is the foundation of modern large language models.

Google Brain