Neural Networks
The model behind modern AI, demystified. From a single artificial neuron to deep networks that learn by adjusting their weights, one small correction at a time.
Neural networks power almost everything people mean when they say "AI" today: image recognition, voice assistants, and the large language models behind ChatGPT. They sound intimidating, but the core idea is remarkably simple. A neural network is a big collection of tiny units, each doing basic arithmetic, connected so that the whole learns from examples by nudging numbers up and down. Understand one neuron and how it learns, and you understand the machine. This page builds it from the ground up, and lets you watch a neuron learn in front of you.
The artificial neuron
The idea was borrowed from the brain. A biological neuron receives signals from many others, and if the combined signal is strong enough, it fires. An artificial neuron copies this in arithmetic. It takes several numbers in, weighs them, adds them up, and passes the total through a function that decides its output.
A neuron has three ingredients:
- Weights: one per input, saying how important that input is. A large weight means the input strongly influences the output.
- Bias: an extra number added in, shifting how easily the neuron fires.
- Activation function: takes the weighted sum and produces the final output.
That is the entire neuron. Everything a neural network does is millions of copies of this one tiny calculation. The intelligence is not in any neuron; it is in the weights, and learning means finding the right weights.
Activation functions
Why not just output the weighted sum directly? Because without a non-linear activation function, stacking neurons would be pointless: any number of linear steps collapses into a single linear step, and the network could never learn curved patterns. The activation adds the bend that makes networks powerful.
| Function | What it does | Note |
|---|---|---|
| Step | Outputs 0 or 1 depending on a threshold | The original perceptron; too abrupt to train well |
| Sigmoid | Squashes any number into a smooth 0-to-1 range | Great for probabilities, can be slow to train |
| ReLU | Outputs the input if positive, else 0 | Simple and fast; the default in deep networks |
From one neuron to a network
One neuron alone is limited. The power comes from connecting many into layers:
- The input layer holds the raw data (for an image, the pixel values).
- One or more hidden layers each take the previous layer's outputs, and every neuron computes its own weighted sum and activation.
- The output layer produces the final answer, such as the probability of each class.
Data flows forward through the layers, each one transforming it a little. Early layers learn simple features (edges in an image), later layers combine them into complex ones (shapes, then faces). Nobody programs these features; the network discovers them by adjusting weights.
The forward pass
Running data through the network to get a prediction is called the forward pass. Each neuron computes its weighted sum and activation, feeds the result forward, and the output layer gives the answer. It is just the neuron calculation, repeated layer by layer.
A neuron with two inputs, using the sigmoid activation:
Scale that up: every neuron in every layer does exactly this, and the final numbers are the network's prediction. At the start, with random weights, the prediction is nonsense. Learning fixes that.
Learning: loss and gradient descent
To learn, the network needs to know how wrong it is. A loss function measures the gap between the network's prediction and the correct answer: a big loss means a bad prediction, zero loss means perfect. Training is simply the job of making the loss as small as possible.
How? With gradient descent, which you have already met in spirit as hill climbing, run downhill. Picture the loss as a landscape where height is error and the position is the current set of weights. Gradient descent works out which direction is downhill (the gradient) and takes a small step that way. Repeat, and the weights slide toward lower and lower loss.
- The gradient tells you, for each weight, whether nudging it up or down reduces the loss.
- The learning rate sets the step size. Too big and it overshoots; too small and it crawls.
- Each step lowers the loss a little, and thousands of steps train the network.
Backpropagation
There is one missing piece. In a deep network, how do you work out the gradient for a weight buried in an early layer, far from where the error was measured? The answer is backpropagation, the algorithm that made neural networks practical.
The idea is to pass the error backward through the network. After the forward pass produces a prediction and the loss is measured, backpropagation starts at the output and works layer by layer toward the input, using the chain rule from calculus to share out the blame: how much did each weight contribute to the error? That gives the gradient for every weight in one efficient sweep, and gradient descent then nudges them all.
Watch a neuron learn
Time to see it happen. Below is a single neuron trying to separate red points from blue ones. It starts with random weights, so its decision boundary (the line) is in the wrong place and the loss is high. Press train and watch gradient descent adjust the weights step by step: the boundary swings into position and the loss curve falls toward zero.
Red and blue are two classes. The line is the neuron's decision boundary; the small curve tracks the loss.
Going deep, and beyond
A network with many hidden layers is called deep, and training such networks is deep learning. Depth is what lets networks learn rich, layered features and tackle images, speech, and language. More layers and more data, with the same forward-and-backward training, is most of the story behind the last decade of AI progress.
The models behind today's chatbots are a special, enormously scaled kind of neural network called a transformer. It uses the same building blocks you just learned, neurons, weights, activations, and gradient descent, arranged in a clever architecture that handles sequences of words. If you want to understand how ChatGPT actually works, this page is the foundation, and the next step is our dedicated guide to transformers.
Key points and practice
- An artificial neuron computes a weighted sum of its inputs plus a bias, then applies an activation function.
- Activation functions (sigmoid, ReLU) add the non-linearity that lets networks learn complex patterns.
- Neurons are connected into layers; data flows forward in the forward pass.
- A loss function measures error, and gradient descent lowers it by nudging weights downhill.
- Backpropagation computes the gradient for every weight by passing the error backward.
- Many layers make a network deep; transformers behind modern chatbots are large neural networks trained this way.
An artificial neuron takes several inputs, each multiplied by a weight that reflects its importance, and sums these products together with an added bias to give a weighted sum. This sum is then passed through an activation function, such as sigmoid or ReLU, which produces the neuron's final output. In short, output equals activation of the weighted sum of inputs plus bias.
Without a non-linear activation function, each neuron just computes a linear combination of its inputs, and stacking linear steps produces only another linear function, no matter how many layers there are. The network could then only separate data with straight boundaries. Non-linear activations let the network bend and combine its transformations, so it can learn complex, curved patterns that a purely linear model cannot.
The network makes a prediction in a forward pass, and a loss function measures how far that prediction is from the correct answer. Backpropagation then passes this error backward through the layers, using the chain rule to compute the gradient, how much each weight contributed to the loss. Gradient descent uses these gradients to nudge every weight a small step in the direction that reduces the loss, scaled by the learning rate. Repeating this forward-and-backward process over many examples gradually lowers the loss and improves the network.