PART 3 · MODULE 3Understanding sequences
RNNs & LSTMs
Recurrent networks for sequential data
Loading module...
Recurrent networks for sequential data
Regular neural networks treat every input independently. They have no sense of "what came before." But order matters. "The dog bit the man" means something very different from "the man bit the dog" — same words, different sequence, different meaning.
For anything involving sequences — text, speech, music, stock prices — we need networks that can remember.
A Recurrent Neural Network (RNN) processes data one step at a time and passes a hidden state forward. Think of it like reading a book: as you read each sentence, you carry the context of everything you have read so far.
At each time step, the RNN takes two inputs: the current word and a summary of everything it has seen before (the hidden state). It combines them and passes the updated summary to the next step.
AN RNN UNROLLED THROUGH TIME STEPS
RNNs have a flaw: they struggle to remember things from far back in a sequence. After processing hundreds of words, the early information gets diluted — like a game of telephone where the message gets garbled.
This is called the vanishing gradient problem. In simple terms: the signal that says "this early information was important" fades to almost nothing by the time it reaches later steps.
Imagine reading a 500-page novel but having no way to take notes. By page 400, you have forgotten most of the details from page 10. That is exactly what happens to a basic RNN.
Long Short-Term Memory (LSTM) networks fix the forgetful reader problem by adding three gates that control what to remember and what to forget. Think of it as giving the reader a notebook.
Decides what old information to erase. Like crossing out irrelevant notes.
Decides what new information to write down. Only the important parts get stored.
Decides what information to share. Like choosing which notes are relevant right now.
LSTM CELL — SIMPLIFIED VIEW OF THE THREE GATES
RNNs and LSTMs were the backbone of sequence modeling for years. But they have a major limitation: they process words one at a time, which makes them slow. In the next module, you will learn about Transformers — the architecture that replaced RNNs by processing entire sequences in parallel.
Memory is essential for understanding sequences. LSTMs solved the "forgetful network" problem by learning what to remember, what to forget, and what to output — giving neural networks a working notebook for the first time.