tokens per second (TPS)

advancedwidespread

It shows how many text fragments (tokens) a language model outputs per second, i.e., how fast its response scrolls before you.

In plain terms

When you ask a question to a conversational AI, the answer does not appear all at once, but rolls out on the screen almost word by word. Tokens per second is the metric that measures this pace. A token is the building block of text: it can be a short word, a word fragment, or a punctuation mark. Due to inflection, Hungarian words often split into multiple tokens.

Think of it like keystrokes per minute for a typist, or kilometers per hour for a car. It does not reveal whether what the model writes is good, only how fast it writes. If a model is slower than you read, you have to wait for it. If it is faster, you barely notice while reading, but for long texts, code, or autonomous agents, the task is still completed sooner.

The metric is important for two reasons. One is the experience: slow response is irritating, and especially annoying with voice assistants. The other is money: for providers, this determines how many users a costly server can serve, and thus the cost of generating a response.

You may encounter it in two senses. One refers to how quickly a single user receives a response; the other refers to how many tokens the entire system generates per second for all users combined. These two figures can differ significantly.

How it works

The model first breaks down your question into tokens and processes them. This is reading the input, which is much faster than writing.
It produces the answer token by token: each new token requires a separate computational step that takes into account everything said so far. This is why output speed is the bottleneck.
Speed is measured by dividing the number of tokens output by the elapsed time. Usually, counting starts from the appearance of the first token, with the initial wait measured separately.
The result depends on many factors: the model size, the chips it runs on, software optimization, and how many people are using the system at the time.
Providers run many requests simultaneously on the same machine. This increases overall system performance, but individual user response times may slow down.

An everyday example

In the evening, during peak hours, you ask a chat AI to write a longer letter, and you see the text flowing slowly, word by word. The next morning, the same request is ready in moments. The model is the same; only the number of tokens written per second changed due to the load.

Who is affected

developerIf you build an application or agent on AI, this determines whether your user experiences the operation as smooth or stuttering. → Code

entrepreneur, executiveSpeed directly affects how long an AI system takes to complete a task and at what cost, so it is one factor in choosing a provider.

employeeIf you use an AI tool in your daily work, the minutes of slow responses add up to noticeable time loss by the end of the day. → Work & HR

everyoneThis determines whether a chat or voice assistant feels like a natural conversational partner.

A common misconception

MythThe more tokens a model writes per second, the better the model.

In realitySpeed says nothing about the quality of the response: a fast model can still be wrong, and the most powerful models are often slower. Above a certain pace, you cannot read the text faster anyway.

Limits and risks

Different models tokenize text differently, so the TPS values of two models cannot be compared precisely.
Advertised figures are produced under ideal conditions; real speed fluctuates depending on load, time of day, and question length.
The metric usually does not include the wait time until the first word appears. Thinking models also generate hidden reasoning tokens before the visible response, so for them this wait can be even longer than the output itself.
With Hungarian text, the same number of tokens means fewer words than with English, so Hungarian responses may feel slower at the same speed.

Similar, but not the same

time to first token – The former measures how long you wait for the response to start, while TPS measures how fast it proceeds afterward.
latency – Latency refers to waiting time: sometimes the duration until the response begins, sometimes until the full response is ready. TPS is only one component of total latency.
context window – The context window specifies how much text the model can consider at once, not how fast it works.
transactions per second – In databases and payment systems, the same TPS abbreviation refers to the number of transactions executed per second and has no relation to tokens.

Related concepts

Broader

inference, performance measurement, throughput

Narrower

output speed, system-level throughput

On the edge

token, latency, time to first token, grafikus processzor, batch processing

A brief history

The metric became widely known with the mass spread of large language models, after ChatGPT's appearance at the end of 2022, when users first saw answers scroll out token by token. Since then, it has become one of the main competitive arenas for chip manufacturers and cloud providers.

Where things stand

At the end of 2026, TPS is the standard metric for comparing models and providers, and independent benchmarking sites report it regularly. As autonomous, multi-step agents become more widespread, its importance keeps growing, because slowness recurs at every step and adds up over the whole task.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

OpenAI unveils always-on Dots agent and GPT-6.1 Sol model at DevDaySep 29, 2026 · Agent Age

Also known as

TPS · tokens per second · token/s · tok/s · tokens per second · tokens per second · output speed · output speed

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.