A/B testing

basicwidespread

A/B testing is an experiment where two versions of something are shown to a randomly split audience, and the better option is decided based on measured results.

In plain terms

Imagine a baker who cannot decide whether to put the cocoa swirl on the left or the right side of the shop window. One week it goes here, the next week there, and they count which week sold more. But between the two weeks, the weather or a holiday could also make a difference. A/B testing is the more precise, online version of this: both variants run at the same time, and visitors are randomly shown one or the other.

Its purpose is to make sure the decision does not hinge on whose voice is loudest in the meeting. They do not ask which button, headline or price people prefer, but look at which version gets more clicks, purchases or sign-ups.

It differs from a simple comparison in two ways. First, the two versions differ in only one aspect. Second, randomization determines which group each subject falls into, so the composition of the two groups is, on average, identical. Therefore, if there is a difference in the outcome, it is likely caused by the change, not by the weather, holidays, or who ended up in which group.

AI applications use it the same way. Two models, two instruction texts (prompts), or two settings of an agent are let loose on real users. They measure which one users are more satisfied with, which one solves more tasks, or which one they return to more willingly.

How it works

First, formulate a hypothesis and a metric: for example, 'the green button brings more purchases,' where the metric is the proportion of buyers.
Two versions are prepared: A is the current one (the control), and B is the one where you changed a single thing.
The system randomly divides visitors into two groups, and everyone consistently sees the same variant for the duration of the experiment.
The test runs until enough data has been collected. How much this is should be calculated in advance. Decisions to stop mid-course are only permissible if a statistical method developed for this purpose is used from the start.
At the end, a statistical test verifies whether the difference is greater than what pure chance could have caused, and only then is the winner rolled out to everyone.

An everyday example

You open an online store, and the 'Add to Cart' button is orange for you, but green on your friend's phone, even though you are viewing the same page. No error occurred: you are both participants in an A/B test, and the store is measuring which color leads to more purchases.

Who is affected

marketerYou can use this to select the better option for ad copy, email subject lines, and landing pages based on data, not intuition. → Marketing

entrepreneur, executiveWith low risk, you can test a new price or offer on a portion of customers before rolling it out to everyone.

developerThis is how you can compare a new feature, model, or agent setting against the old one in live traffic with measurable results. → Code

everyoneYou are participating in such experiments on almost every major website and application without your knowledge, which is why you might see something different from your friend.

A common misconception

MythIf variant B performs better after a few days, the test can be safely stopped, as the winner is clear.

In realityWith little data, the difference is often just random fluctuation, and tests stopped too early frequently declare a false winner. The required sample size and duration must be fixed in advance. The other option is to use a method that inherently allows for mid-course checks.

Limits and risks

On a low-traffic site, it can take weeks or months to produce reliable results, and even then, small differences cannot be detected.
It shows which variant performs better, but not why; understanding the reasons requires interviews or other research.
It optimizes for short-term metrics (such as clicks), which can harm trust or satisfaction in the long run.
Testing different prices or content influencing emotions raises ethical and data protection issues, because participants are typically unaware of the experiment.

Similar, but not the same

Multivariate testing – There, combinations of multiple variants of several elements are examined, whereas an A/B test compares a single change with the original.
Personalization – Personalization intentionally shows different things to different people based on their characteristics, while A/B testing assigns randomly to find out what works generally.
market research (survey, focus group) – Market research records what people say; an A/B test records what they actually do.

Related concepts

Broader

Controlled experiment, data-driven decision-making, Conversion optimization

Narrower

Split URL testing, A/B/n testing

On the edge

Multivariate testing, multi-armed bandit algorithm, Conversion rate, Statisztikai szignifikancia, Personalization, Recommendation system, Model evaluation

A brief history

The method originates from randomized controlled trials. Their foundations were laid by the British statistician Ronald Fisher in the 1920s during agricultural experiments. Internet companies began applying them massively in the early 2000s to develop websites and search engines. One of Google's first known such tests in 2000 was about how many results to display on a page.

Where things stand

At the end of 2026, this is a basic tool of digital product development and marketing, built into most advertising and newsletter systems. It is also established practice in AI development for comparing models, prompts and agents live, and the variants are increasingly often generated by generative AI.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

Google Ads adds generative AI for Performance Max video, new AI features and AI Max updateAug 17, 2026 · Marketing

Also known as

A/B teszt · A/B testing · AB teszt · A/B testing · A/B test · split teszt · split testing · bucket testing

Sources

Wikipedia: A/B testing

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.