PART 3 · MODULE 5How AI learns
Training Neural Networks
Backpropagation, optimization, and more
Loading module...
Backpropagation, optimization, and more
Training a neural network means adjusting millions of internal numbers (called weights) until the network gets good at its task. It is a loop: predict, measure the error, adjust, and repeat — millions of times.
The loss function measures how wrong the model is. Think of it as a grade on a test: a high loss means the model is making big mistakes, a low loss means it is getting things right.
The entire goal of training is to minimize this number. Every adjustment to the network is aimed at pushing the loss lower.
Gradient descent is the algorithm that adjusts the weights. Imagine you are lost in a foggy mountain landscape and you want to reach the lowest valley. You cannot see far, but you can feel the slope right where you stand. So you take a step in the steepest downhill direction, then feel the slope again, and repeat.
GRADIENT DESCENT — ROLLING DOWNHILL TO FIND THE LOWEST POINT
The learning rate controls how big each step is during gradient descent. Get it wrong and training fails:
You overshoot the valley and bounce around wildly. The model never converges.
You inch forward so slowly that training takes forever and may get stuck.
You make steady progress toward the lowest point. The model learns efficiently.
Backpropagation is how the network figures out which weights to adjust and by how much. It works backwards from the output: if the answer was wrong, which neurons in the last layer were most responsible? Which neurons in the layer before that contributed to those?
It is like tracing back through a chain of mistakes. Each layer gets a signal that says "here is how much you contributed to the error — adjust accordingly."
One full pass through the entire training dataset. Models typically train for many epochs — going through the data multiple times to refine their understanding.
A small chunk of the training data processed at once. Instead of learning from all examples simultaneously, the model processes small batches (e.g., 32 or 64 examples at a time).
When the model memorizes the training data instead of learning general patterns. Like a student who memorizes test answers but cannot solve new problems.
When the model has not learned enough. It performs poorly on both training data and new data — it did not study enough.
Randomly turn off neurons during training. This forces the network to not rely on any single neuron — like studying with different flashcard sets each time.
Training requires millions of math operations in parallel. GPUs (graphics cards) were designed for exactly this. That is why AI companies spend billions on GPU clusters.
Start from a model that was already trained on a huge dataset, then fine-tune it for your specific task. Like learning Spanish when you already know Italian — much faster than starting from scratch.
Artificially expand your training data by applying transformations — flipping images, adding noise, rephrasing text. More data usually means better models.
Training is just an optimization loop — predict, measure the error, adjust the weights, and repeat millions of times. Every AI model you use, from ChatGPT to self-driving cars, was born from this same simple cycle. The magic is not in the algorithm — it is in the scale of data and compute thrown at it.