How Neural Networks Learn: Layers, Weights, Training

A neural network is a computer model made of many small calculating units, called artificial neurons, arranged in layers. It does not follow hand-written rules. Instead it starts with arbitrary numbers, compares its answers with known correct answers, and nudges those numbers until the answers improve. This article explains the parts and the training loop in plain terms.

What one artificial neuron does

An artificial neuron is loosely inspired by the nerve cells of a brain, but it is just arithmetic. It receives several numbers as input. Each input is multiplied by a weight, which says how strongly that input matters. The neuron adds up the results and then adds one more number called a bias, which shifts the point at which the neuron becomes active.

The total then passes through an activation function. This is a simple rule that turns the sum into the neuron's output. Common choices named by IBM include tanh, sigmoid and ReLU. Without this step the whole network would collapse into one big weighted sum, and it could not learn curved or complicated patterns.

Layers: input, hidden and output

Neurons are grouped into layers. The input layer holds the raw data, for example the brightness of each pixel in a photo. One or more hidden layers sit in the middle and turn the data into new representations. The output layer gives the answer, such as a probability for each possible label.

Signals move forward from layer to layer. Wikipedia notes that a network with at least two hidden layers is usually called a deep neural network, which is where the term deep learning comes from. Every connection between neurons has its own weight, so even a modest network can hold millions of these adjustable numbers. They are often called parameters.

How training works

Training repeats four steps over the examples it is given.

• Forward pass: the inputs flow through the network and produce a prediction.
• Loss: a loss function measures how far the prediction is from the known correct answer.
• Backward pass: an algorithm called backpropagation works out how much each weight contributed to the error.
• Update: a method such as gradient descent changes each weight a small step in the direction that reduces the error.

The size of each step is set by the learning rate. According to Wikipedia, a higher learning rate can train faster but may end with lower accuracy. Training goes through the data many times, and the loss usually falls as the weights settle.

Overfitting: learning too well

A network with a great many parameters can memorise its training examples instead of learning the general pattern. This is called overfitting. It then performs well on data it has seen and poorly on new data. IBM describes this as part of the trade-off between bias and variance, and it is why developers test on examples that were held back from training.

Common kinds of network

• Feedforward networks: information flows one way, input to output. Multilayer perceptrons are the basic example.
• Convolutional neural networks (CNNs): built for grid-like data such as images, using small filters that detect edges and shapes.
• Recurrent neural networks (RNNs): include feedback loops so they can carry information from one step to the next, which suits speech and time series.
• Transformers: use a mechanism called attention. Introduced in the 2017 paper "Attention Is All You Need", they now underlie many large language models and have replaced recurrent networks for many sequence tasks.

A short history

The idea is older than most people expect. Wikipedia dates the first mathematical model of neural networks to 1943, by McCulloch and Pitts. Frank Rosenblatt described the perceptron in 1958, with funding from the US Office of Naval Research. Backpropagation in its modern form appeared in a 1970 master's thesis by Seppo Linnainmaa, was applied to neural networks by Paul Werbos in 1982, and was popularised in 1986 by Rumelhart and colleagues.

A turning point for image recognition came in October 2012, when AlexNet, by Krizhevsky, Sutskever and Hinton, won the large ImageNet competition by a wide margin over methods that did not use deep networks.

What a network does not do

A trained network is a fixed set of numbers that maps inputs to outputs. It does not understand the data the way a person does, and its quality depends heavily on the examples it was trained on. The sources used here describe the mechanism, not any particular product, and real systems add many more techniques around this core. This is general information, not a guide to building or buying a system.

Sources

• Neural network (machine learning), Wikipedia)
• What is a neural network?, IBM