OSWorld

advancedwidespread

OSWorld is a test suite measuring how well an AI agent can independently perform everyday computer tasks on a real operating system with real software.

In plain terms

Imagine the practical driving test. There, they do not quiz you on traffic rules; instead, they put you in a car and you must show you can get from A to B. OSWorld is the same for AI agents: the agent is given a test computer and must actually perform a task on it.

The tasks are like those you do in the office or at home: filling out a spreadsheet, resizing an image, saving an email attachment, rewriting settings in the browser. The agent 'looks' at the screen and works with a mouse and keyboard as a human would.

It became important because it provides a comparable number for where computer-operating agents stand. When an AI company advertises that its model 'can already use your computer', it often refers to the OSWorld result.

How it works

Researchers launch a virtual machine, an isolated "computer within a computer," running a real operating system and real programs (such as a browser, office suite, or image editor).
The agent receives a text instruction, for example: "Create a report from this table and save it as PDF."
The agent receives screenshots from the machine (in some settings, also a text description of the screen content) and makes step-by-step decisions: where to click, what to type, what to scroll. After each step, it sees the new state.
At the end, a verification program checks the machine's actual state: was the file created, is the content correct, were the settings changed. What matters is whether the task is done, not the agent's own report.
The results of the 369 tasks are turned into a percentage success rate, which is usually compared with the roughly 72 percent achieved by human testers.

An everyday example

You read a news report that a new AI model 'broke a record on OSWorld' and now handles computers better than humans. This means that on a laboratory test machine, it solved a larger portion of the released office and browser tasks than previous models, and more than the human testers, who achieved roughly 72 percent. It does not mean it will be able to handle everything on your corporate computer.

Who is affected

employeeOSWorld results indicate how quickly we are approaching the point where agents can also perform routine screen-based office work. → Work & HR

entrepreneur, executiveIf you are buying a computer-operating agent, this is one of the most frequently cited metrics for comparing vendor offerings.

developerAn open test environment runnable by anyone, where you can try out your own agent and compare it with others. → Code

A common misconception

MythIf an agent reaches human level on OSWorld, any computer work can be entrusted to it.

In realityThe test contains a few hundred pre-written, relatively short tasks in a clean lab environment. A real workplace machine with unique programs, permissions, and unexpected situations is much more complex, and there, a mistake has a real cost.

Limits and risks

The success rate only shows whether a task was completed, not how much time and money it took: an agent often works much slower than a human.
Since the tasks are public, developers may, intentionally or not, tune their systems to them, so a high score does not necessarily indicate general proficiency.
Results can only be compared under identical settings (for example, how many steps the agent can take, which test version was measured), but this is often omitted from news reports.
The test barely measures security: it does not reveal whether the agent can be deceived by a malicious website or accidentally delete something important.

Similar, but not the same

Computer use – Computer use is the ability of an agent to handle a mouse and keyboard, while OSWorld is the exam used to measure this capability.
SWE-bench – SWE-bench measures whether an agent can fix real programming bugs in code, while OSWorld measures whether it can use programs through the screen.
WebArena – WebArena tests tasks performed only on websites within a browser, while OSWorld tests the entire desktop computer, including multiple programs and files.

Related concepts

Broader

Benchmark, AI-agent, agent evaluation

Narrower

OSWorld-Verified

On the edge

Computer use, SWE-bench, WebArena, browser agent, virtual machine

A brief history

OSWorld was introduced in 2024 by a research group led by the University of Hong Kong. At that time, the best AI system solved roughly 12 percent of tasks, while humans solved about 72 percent. In 2025, OSWorld-Verified was released, an improved version with more reliable tasks.

Where things stand

By the end of 2026, it is one of the most cited benchmarks for computer-operating agents; AI companies regularly publish their results on it. According to developers, the best models reached, and in some cases exceeded, the approximately 72 percent human-level performance in 2026. Consequently, the test is increasingly unable to distinguish among the top performers, and researchers are working on successors consisting of more difficult and longer tasks.

Status: 2026 November

Also known as

OSWorld · OS World · OSWorld benchmark · OSWorld-teszt · OSWorld-Verified · OSWorld benchmark

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.