What Are Neural Networks?

The café screen says your order will take ten minutes. A second barista starts work; the estimate falls to seven.

Has the system learned something?

Not necessarily. A model can give a different answer simply because it received different information. Training the network means changing its adjustable parameters.

To see the difference, we will open up a tiny neural network and follow every calculation. It is deliberately small enough to inspect, and deliberately invented: these numbers explain a mechanism, not the waiting time at a real café.

A neural network is a calculation with adjustable parts

A neural network processes numerical inputs through connected computational units, often arranged in layers. Its parameters—including weights and biases—control how those calculations behave. Training adjusts parameters so the network performs a chosen task better. [1][3]

The word “neuron” names a mathematical unit here. It should not be taken to mean that the network is a miniature human brain.

In a common kind of unit, each input is multiplied by a weight, the results are added, and a bias is added to shift the total. An activation function then transforms that result. [1][2]

That sounds abstract until we give each part a number.

Meet our very small café model

Our model takes two inputs:

• q: the number of orders ahead of yours;
• b: the number of baristas working.

Its output is an estimated wait in minutes. Between the inputs and the output sits one hidden layer with two units, A and B. “Hidden” means intermediate: these are neither the supplied inputs nor the final output.

We have chosen the following parameters by hand to make the calculation easy to follow:

In compact notation:

A = max(0, 2q − b − 1)
B = max(0, q − 2b)
Estimated wait = 2A + B + 1

The numbers multiplying inputs are weights. The extra −1 in A and +1 at the output are biases; B has a bias of zero. A bias in this mathematical sense is an adjustable offset, not a statement about social bias.

Start with three orders ahead and one barista:

That is a forward pass: inputs move through the calculations to produce an output. Using a model in this way to make a prediction is inference.

What does an activation function actually do?

The “keep it if positive, otherwise use zero” operation is called ReLU, short for rectified linear unit. For example, ReLU turns −1 into 0, leaves 3 as 3, and returns 0 for an input of 0. It is one common activation function, not the only choice. [2]

Without nonlinear operations, stacking these weighted sums would just produce another weighted sum. Nonlinear activation functions let the network express relationships that a purely linear stack cannot. [1][2]

In our example, B contributes nothing when its calculation falls below zero. That changes how the output responds to the inputs. You will see it happen when we add a barista.

The values A and B are activations produced for this particular input. They are not extra parameters that training has stored. The weights and biases are the adjustable settings; the activations are results calculated using them.

Two ways to change the answer

Try It with Byte

Byte asks: “The estimate changed. Did the café change—or did the model?”

Change the café input

Keep every weight and bias exactly as it was. There are still three orders ahead, but now two baristas are working.

• A becomes 2 × 3 − 2 − 1 = 3.
• B’s calculation becomes 3 − 2 × 2 = −1, so ReLU makes B 0.
• The output becomes 2 × 3 + 0 + 1 = 7 minutes.

The prediction changed, but the model did not learn. We used the same function with a different input.

Nor have we established that a second barista really causes a three-minute improvement. We have only established what this invented model calculates.

Change the model

Now reset to the original situation: three orders, one barista. A is 4, B is 1, and the original estimate is ten minutes.

Suppose a fictional training record says the actual wait for that example was eight minutes. That observed value is the training target, or label. The prediction is two minutes too high.

For this demonstration, hold every parameter fixed except the weight connecting A to the output. Lower that weight from 2 to 1.5:

New estimate = 1.5 × 4 + 1 + 1 = 8 minutes.

This time the input stayed the same and a parameter changed. That is the kind of change training makes.

The last two rows are separate changes from the baseline, not consecutive steps.

How does training choose an update?

In a real training process, someone does not usually inspect every prediction and hand-edit the weights. A loss function turns the mismatch between predictions and targets into a number to reduce.

Backpropagation computes how that loss changes with respect to the parameters: its gradients. An optimiser uses those gradients to update parameters. Backpropagation computes the information for the update; it is not itself the entire training process. [3][4]

For our one adjustable output weight, reducing it reduces the overestimate. A suitably sized gradient step can produce the 2-to-1.5 update shown above. We have frozen the other parameters to isolate the mechanism; ordinary training can update many parameters across multiple layers.

The size of a step matters too. A learning rate helps control how far an update moves. Training repeatedly runs predictions, measures loss and adjusts parameters, usually using batches of examples. [4]

Getting our single invented example exactly right proves very little about future orders. The model could fit examples it has seen and still predict badly on new ones. Evaluating on separate data helps check generalisation—performance beyond the training examples. [5]

Our café model also ignores order complexity, equipment, interruptions and many other factors. Its transparent arithmetic should not be mistaken for a validated waiting-time system.

What changes in a larger network?

The same basic distinction survives: parameters define the calculation; activations depend on the input; training updates parameters; ordinary inference uses them without a learning update.

Larger networks can contain many more units and layers, and architectures can arrange their operations differently. Our two-unit hidden layer is one small feed-forward example, not a diagram of every network ever built.

Consider the existing everyday example of reading a handwritten digit. The inputs can represent image pixels, intermediate layers transform those numbers, and the output can contain scores for the possible digits. The network has to learn useful distinctions from examples rather than receive a handwritten rule for every way someone draws a “7”. [1]

An output’s meaning depends on how the model is designed. Our output is a number interpreted as minutes. A digit classifier can have several class scores. A number leaving a network is not automatically a confidence percentage.

Weights also require context. A weight of 2 does not universally mean an input is “twice as important”: input units, scaling, other connections and nonlinear operations all affect the result. In our tiny network we can trace a weight’s role exactly. Reading one weight in a large network rarely explains the whole answer.

What this small network lets you see

The term “neural network” can make the mechanism sound distant. Here, you have seen its essential pieces doing concrete work:

• Inputs supply numbers for the current case.
• Weights and biases shape calculations.
• Activation functions transform intermediate results.
• Layers connect those operations into a model.
• Inference produces an answer; training changes parameters using an objective and examples.

The ten-minute estimate became seven because the input changed. It became eight in our learning example because a parameter changed. Different answers do not, by themselves, show that a model learned.

That distinction is useful far beyond a café. When an AI system responds differently after you add information, ask what actually changed: its current input, its stored parameters, or another part of the surrounding application.

Continue learning

Previous: What Is Deep Learning? — what layered representations can accomplish, and why the task still matters.

Need the wider foundation? What Is Machine Learning? explains training examples, objectives and evaluation.

Next: What Are Large Language Models? — how a neural-network model uses context to generate language.

Explore an architecture: What Is a Transformer in AI?.

Explore the AI Fundamentals learning path.

Sources

1. Google Machine Learning Crash Course, Nodes and hidden layers. Weighted calculations, intermediate layers and the need for nonlinearity.
2. Google Machine Learning Crash Course, Activation functions. ReLU and nonlinear transformations.
3. Google Machine Learning Crash Course, Backpropagation. Gradients through a network.
4. PyTorch, Optimizing Model Parameters. Loss, learning rate and the distinction between computing gradients and taking an optimiser step.
5. Google Machine Learning Crash Course, Overfitting. Training performance versus generalisation.

The café network is an original, hand-specified mathematical illustration. It has not been fitted to café data or evaluated as a real service.