gradient descent (GD)

advancedwidespread

Gradient descent is the method by which an AI model fine-tunes its internal settings in small steps, gradually reducing its error.

In plain terms

Imagine standing on a mountainside in thick fog, wanting to reach the valley. You cannot see the path, but you feel the slope under your feet. You take a step, feel the slope again, and take another step. Gradient descent works exactly like this, except the mountain is the model's error: the lower you go, the less the model errs.

An AI model consists of a vast number of adjustable numbers, called parameters; in large language models, these number in the billions. Training means adjusting these numbers until the model gives good answers. Gradient descent determines which number to move, in which direction, and by how much.

It does not try blindly, nor does it calculate the perfect solution in advance. It only looks at its immediate surroundings and steps in the direction of the steepest slope. This makes it simple and easily automatable, and allows training of huge models.

How it works

The model starts with random settings, so at first it is wrong about almost everything.
It receives a few training examples, answers them, and a metric (the loss function) shows how much it erred.
They calculate the gradient: for each parameter, it shows how much a small shift would increase or decrease the error.
Every parameter is nudged slightly in the direction that reduces error. The step size is determined by the learning rate: if it is too large, the model skips over the valley; if too small, training becomes very slow.
This is repeated millions of times with new examples until the error barely decreases further.

An everyday example

When your phone keyboard predicts the word you want to type, a model trained this way works in the background. Translators, image recognizers, and chatbots are all built by repeating this step-by-step error reduction for hours, weeks, or even months, depending on the model's size.

Who is affected

developerWhen training or fine-tuning a model, the choice of learning rate and optimizer directly determines whether training succeeds. → Code

entrepreneur, executiveMost of the time and cost of model training is the machine time spent on this many-times-repeated calculation, which is how you can see why a custom model is expensive.

parent, student, teacherA good example showing that machine 'learning' is actually error-reducing calculation, also demonstrating the practical benefit of differentiation. → Education & Family

everyoneAlmost every AI tool you use today was trained with this method, helping you realistically understand its capabilities and limitations.

A common misconception

MythGradient descent finds the best possible model.

In realityIt only shows in which direction the error decreases in the immediate vicinity; it does not guarantee that it stops at the best solution. With a poorly chosen step size, the error may even temporarily increase. In practice, it reaches a "good enough" point, and the result depends on the starting point and the settings.

Limits and risks

It can get stuck or slow down significantly in a local minimum, flat plateau, or saddle point, where no better direction is visible from the immediate surroundings.
It is sensitive to the learning rate: if poorly chosen, training will collapse or barely progress.
With large models, enormous computing capacity and energy are required because the step must be repeated countless times.
It only reduces the error you can measure: if the training data is biased or the goal is poorly defined, the model learns the wrong goal very well.

Similar, but not the same

backpropagation – Backpropagation calculates the gradient in the neural network; gradient descent then adjusts parameters based on this.
reinforcement learning – Reinforcement learning is a learning scenario (a trial-and-error system based on rewards, such as an agent), while gradient descent is a computational procedure often used within it.
fine-tuning – Fine-tuning is the further training of a pre-trained model on your own data; gradient descent is the technical method by which this training occurs.

Related concepts

Broader

optimization, machine learning, model training

Narrower

stochastic gradient descent, mini-batch training, Adam optimizer

On the edge

backpropagation, loss function, learning rate, parameter, Neural network

A brief history

The method was described by French mathematician Augustin-Louis Cauchy in 1847 for solving systems of equations. The foundations of the stochastic variant, which works with random samples, were laid by Herbert Robbins and Sutton Monro in 1951. It became widespread in neural network training after the 1986 backpropagation study (Rumelhart, Hinton, Williams).

Where things stand

At the end of 2026, practically every large AI model, including language models and image generators, is trained with some advanced variant of gradient descent. Research focuses mainly on faster, less energy-intensive variants.

Status: 2026 November

Also known as

gradient descent · gradient descent · gradient method · gradient descent · steepest descent method · gradient descent · steepest descent

Sources

Rumelhart, Hinton, Williams: Learning representations by back-propagating errors (Nature, 1986)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.