throughput

advancedwidespread

Throughput indicates how much work a system performs in a given time, for example, how many requests or text segments it processes per second.

In plain terms

Think of a highway toll gate. The question is not how fast a car drives through it, but how many cars it allows through per hour in total. This is throughput: the system's total "traffic" over a given time.

For AI systems, this is usually measured in tokens (text fragments) or requests: how many tokens the system generates per second for all users combined, and how many requests it fulfills per minute. This determines whether you get a response during peak times and how much the service costs.

Different from response time. Response time shows how long you wait for your own question. Throughput shows how much the system can handle in total. A system can be lightning-fast for a single person while being insufficient for a thousand.

How it works

Requests arrive at the provider's servers, and if there is no available capacity, they queue up.
The system bundles multiple requests and runs them together on accelerator cards (GPUs) because this allows the same hardware to perform more work. Modern systems also expand the bundle on the fly: as soon as one request is completed, a new one can take its place.
The model generates responses token by token. Throughput is defined by how many tokens are produced per second across all running requests combined, including both input and output text.
The operator weighs the trade-off: larger batches increase overall throughput, but individual users may wait longer, so the operator seeks a balance between the two.
If traffic consistently exceeds capacity, the provider either limits requests or brings additional machines online.

An everyday example

On Monday morning at nine, you ask a chatbot to summarize a long report, and the response trickles slower than usual, or you get a 'too many requests' message. In this case, you have hit the limit of the service's throughput: more people are asking for something from it at once than it can handle.

Who is affected

developerThis determines how many users your application can support and how many agents you can run simultaneously. → Code

entrepreneur, executiveThroughput determines the size of the server farm and the monthly bill required behind an AI service.

employeeDuring peak times, this determines whether the workplace AI tool responds immediately or causes waiting and restrictions. → Work & HR

A common misconception

MythIf a system has high throughput, it will also answer my question quickly.

In realityHigh throughput only means that the system performs a large total amount of work. Individual requests can still be slow, because requests are often delayed and batched together precisely for the sake of overall performance.

Limits and risks

Numbers provided by manufacturers and service providers are peak values measured under ideal conditions; under real load, you typically get less.
Throughput says nothing about the quality of the answer: a model that produces a lot quickly can still be wrong.
Token/second values can only be compared cautiously between models because models split text differently and request length also distorts results.
Most providers limit the number of requests and tokens per minute by subscription tier, so your own quota may be tighter than what the system can handle.

Similar, but not the same

latency (response time) – Latency measures how long you wait for the result of a single request, while throughput measures how many requests the system completes in total within a given time.
bandwidth – Bandwidth is the theoretical upper limit of a network connection, while throughput is what actually passes through it.
capacity – Capacity shows how much resource is available, while throughput indicates how much of that results in completed work.

Related concepts

Broader

performance measurement, AI infrastructure, scalability

Narrower

tokens per second, requests per second, batch processing

On the edge

latency, inference, GPU, rate limit, token

A brief history

The term moved from manufacturing and telecommunications into computing, where it became a key metric for networks and servers. Since 2019, the MLPerf Inference industry benchmark has also compared the serving performance of AI systems, partly based on throughput.

Where things stand

At the end of 2026, throughput is one of the basic metrics of AI services. Providers specify it in tokens per second and per-minute limits. With the spread of long-running and parallel agents, it is increasingly the bottleneck and a significant cost factor.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

OpenAI unveils always-on Dots agent and GPT-6.1 Sol model at DevDaySep 29, 2026 · Agent Age

Also known as

throughput · throughput · throughput · Throughput · tokens per second · tokens per second · TPS · requests per second

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.