latency

basicwidespread

Latency is the time between sending your request and receiving the response – in other words, how long you have to wait for the machine.

In plain terms

If you order at a restaurant, it matters when the food arrives. Latency is the same wait in the digital world: the time elapsed from sending your question to the appearance of the answer. It can be a few milliseconds, but for a complex AI task, even several minutes.

With AI, two things are worth watching separately. One is when the first word appears, the other is when the full answer is ready. In a restaurant, it feels different if the waiter brings the bread in two minutes than if no one looks at you for half an hour, even though the main course arrives at the same time in both cases.

Latency is not about quality but about speed. A larger, smarter model often responds more slowly, while a smaller one is faster but more superficial. So you always have to weigh whether speed or thoroughness matters more in the given situation.

How it works

Your request travels over the network to a data center; the farther the server is and the weaker your connection, the longer the journey takes.
If many people use the service simultaneously, your request queues until computing capacity becomes available.
The model processes the input: the longer the text or attached document, the later the response begins.
The response is generated piece by piece, token by token, so longer answers take more time; most applications continuously display what the model has already written, so you do not have to wait for the end.
If an agent is working on a task, it takes multiple steps in sequence—searching, calling tools, verifying—and these delays add up.

An everyday example

You ask a chatbot a question, and for a second or two you only see blinking dots before the text starts: this is latency. You feel the same when a phone voice assistant stays silent awkwardly long after your question before speaking.

Who is affected

everyoneWaiting time determines whether you perceive an AI tool as smooth or annoyingly slow.

developerIt must decide which model, input length, and technical solution fit within the acceptable response time. → Code

entrepreneur, executiveIn customer service or sales AI solutions, slow responses mean losing customers, while faster service may cost more.

industrial worker, engineerIn the control of machines, robots, and vehicles, latency is not a matter of convenience but of safety. → Industry

A common misconception

MythIf the AI responds slowly, your internet connection is definitely weak.

In realityMost of the waiting time usually occurs on the server: the model is computing, or your request is queued. The network is only one factor, often a smaller one.

Limits and risks

Latency fluctuates strongly: at peak times, the same question may take multiple times longer to process, so the average value tells little about the worst moments.
Speed has a cost: faster responses often require a smaller, less thorough model or more expensive hosting.
Long-thinking models and multi-step agents are intentionally slower because they do more work for better results.
A fast answer does not mean a correct one: from the speed you can learn nothing about the reliability of the content.

Similar, but not the same

throughput – Throughput shows how many requests a system can handle in a given time, while latency shows how long one must wait for a single request.
bandwidth – Bandwidth shows how much data can be transferred per second, while latency indicates how long it takes for data to reach its destination – latency can be high even on a high-bandwidth connection.
response time – In everyday speech and AI services, these terms usually mean the same thing. In network technical texts, however, latency sometimes refers only to the time until the response begins, while response time covers the duration until the full response is ready.

Related concepts

Broader

performance, AI infrastructure

Narrower

time to first token, network latency, ping

On the edge

inference, token, streaming, edge computing, throughput

A brief history

The term originates from telecommunications and computer networks, where it denoted the time taken for a signal to travel. The general public learned it through the "lag" and "ping" terms in online games, and since the launch of ChatGPT in 2022, it has become one of the most important metrics for AI services.

Where things stand

At the end of 2026, latency is a key metric for AI services: providers compete with dedicated fast model variants and accelerated serving. Voice assistants already respond at near-natural speed, but for long-running agents, waits measured in minutes have become common.

Status: 2026 November

In the news

Appears in 43 Agent Age stories; the curve shows weekly occurrence in the news magazine.

OpenAI builds new Git backend Pando after Codex agents drive commit surgeNov 4, 2026 · Code
HDFC and Kotak: AI agent safeguards have not kept pace with data access, The Ken reportsNov 3, 2026 · Money
JPMorgan, Amazon and Bank of America restrict external AI agents' accessNov 2, 2026 · Money
ECB's Lane: AI is a significant driver of euro-area growth and lendingOct 30, 2026 · Money
AI political ad spending in US midterms tops $137 million as regulation lagsOct 26, 2026 · Marketing
Banks hand more decisions to AI as governance lags behindOct 26, 2026 · Money
Central and Eastern European banks, including Hungarian ones, catch up on AI useOct 26, 2026 · Money
U.S. manufacturing rebound is lopsided, led by AI and energy-boom industriesOct 26, 2026 · Ipar

Also known as

latency · latency · latencia · latency · response time · lag · time to first token · TTFT

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.