Recurrent neural network

Recurrent neural networks gave artificial intelligence the ability to remember—to process sequences where context matters, where what came before shapes what comes next. From early associative memories to the language models transforming computing, RNNs established that temporal patterns could be learned through connection weights.

The adjacent possible required understanding how biological neural networks process time. While feedforward networks could only respond to instantaneous inputs, real brains maintain state—context from the past influences present processing. The challenge was creating artificial networks that could similarly incorporate history without explicitly programming every temporal relationship.

John Hopfield provided the breakthrough in 1982. Then a physicist at Caltech, Hopfield introduced networks where neurons connected to each other in loops, not just in layers. These Hopfield networks could store and retrieve patterns like associative memories—present a corrupted input, and the network would settle into the nearest stored pattern. Crucially, the network maintained internal state that evolved over time. Hopfield's work reignited interest in neural networks after the AI winter of the 1970s.

The 1986 publication of 'Learning Internal Representations by Error Propagation' by Rumelhart, Hinton, and Williams—rediscovering Werbos's 1974 backpropagation algorithm—transformed what recurrent networks could learn. Instead of hand-programming connection weights, networks could adjust weights automatically to minimize prediction errors. Backpropagation Through Time (Williams and Zipser, 1989) extended this to arbitrary sequence lengths.

Michael Jordan's 1986 recurrent architecture fed outputs back as additional inputs—letting networks learn from their own predictions. Jeffrey Elman's 1990 'Simple Recurrent Network' (Elman network) fed hidden layer activations back as context units, demonstrating that networks could learn grammatical structure from raw text. These architectures showed that temporal patterns in language, music, and motor control could emerge from connection weights alone.

Path dependence shaped the field. Early successes with language processing established RNNs as the default architecture for sequential data. But standard RNNs suffered from vanishing and exploding gradients—information from distant past inputs faded or exploded as it propagated through many time steps. This limitation drove Hochreiter and Schmidhuber's 1997 invention of Long Short-Term Memory (LSTM), which solved the gradient problem and dominated sequence modeling for two decades.

The cascade was transformative. RNN variants powered machine translation, speech recognition, and text generation. Google Translate switched to RNN-based neural machine translation in 2016. Voice assistants used RNNs for speech recognition. The architectures eventually yielded to Transformers in 2017—but Transformers themselves were born from the limitations researchers discovered while pushing RNNs to their limits.