SWE-bench

advancedwidespread

SWE-bench is a test that measures how well an AI model or coding agent can perform programming work independently by fixing real software bugs.

In plain terms

Imagine testing an auto mechanic apprentice not with a test sheet but by pushing a genuinely broken car in front of them: find the fault, fix it, and ensure the engine starts at the end. SWE-bench does the same with AI, only with software instead of cars.

The tasks are not made-up school examples. They come from real bug reports and minor development requests for well-known open-source programs, which were previously solved by human developers. The AI receives the full code and the problem description, and must write the fix itself.

It became important because it shows with a single percentage figure how much of the tasks a model solved. Companies quote this number when announcing a new model, and journalists compare it as well.

It differs from earlier programming tests, which gave short, independent challenges. Here, you must navigate a large, live project consisting of even thousands of files, as in a real workplace.

How it works

Researchers collected bug reports from popular open-source Python projects that later received accepted fixes.
The AI receives the project code in its pre-fix state and the bug report text; it does not see the original solution.
The model or agent reviews the code, locates the error, and prepares the modification.
The project's automated tests are run on the modified code, including those that previously failed due to the error and those that were previously passing.
A task counts as solved only if the bug is gone and nothing else has broken in the process. The result is the share of tasks solved.

An everyday example

When you read in a news report that a new AI model "performed above 80 percent on the SWE-bench Verified test", it means that out of 500 verified, real software bugs, it independently fixed more than 400 in a way that the tests also passed.

Who is affected

developerThis is one of the most frequently cited figures when deciding which coding agent or model to try. → Code

entrepreneur, executiveSuppliers argue with this number, so it is worth knowing what it proves and what it does not before subscribing to an AI tool.

jobseeker, new graduateThe rapid improvement in test results indicates which routine programming tasks are increasingly being taken over by AI. → Work & HR

bank customer, investorIn AI company announcements, this is often the key performance metric that also influences their market perception. → Money

A common misconception

MythIf a model achieves 80 percent on the SWE-bench, it can perform 80 percent of programming work.

In realityThe test measures well-defined, automatically verifiable bug fixes, mainly in Python. Planning, coordination, clarifying vague requirements, and long-term maintenance are excluded.

Limits and risks

The tasks originate from public code, so models may have encountered the solutions during training, which can upwardly bias the results.
Both the original version and Verified are built solely on Python projects, so they say little about other languages or about enterprise, closed systems.
The result depends on the tools and settings built around the model, so figures reported by different companies are not always comparable.
Success is indicated by passing the tests, which does not guarantee that the fix is neat, secure, or maintainable in the long term.
The verification tests are not flawless either: for some tasks they reject a correct solution, or expect a detail that the task description does not state.

Similar, but not the same

HumanEval – HumanEval provides short, standalone programming challenges, while SWE-bench requests bug fixes in full, real-world projects.
SWE-bench Verified – This is a narrower part of SWE-bench consisting of 500 tasks reviewed by humans to ensure the tasks are clear and solvable.
coding agent – The coding agent is the tool that does the work, while SWE-bench is the exam used to measure it.

Related concepts

Broader

Benchmark, AI model evaluation, Ranglista

Narrower

SWE-bench Verified, SWE-bench Lite, SWE-bench Multimodal

On the edge

coding agent, AI-agent, HumanEval, Training data contamination, Large language model

A brief history

SWE-bench was introduced in autumn 2023 by researchers at Princeton University in collaboration with the University of Chicago. The original dataset consisted of 2,294 tasks across 12 Python projects, and the best model at the time solved barely 2 percent of them. In August 2024, OpenAI and the original authors jointly released SWE-bench Verified, a human-verified set of 500 tasks that became the industry's primary benchmark.

Where things stand

At the end of 2026, this remains the best-known benchmark for coding AIs, but leading models already achieve results around or above 80 percent on the Verified version, making the test barely distinguish between them. Due to saturation, data contamination, and flawed tests, the industry is increasingly turning to new, harder tests (such as SWE-bench Pro). In early 2026, OpenAI indicated that it no longer considers Verified results to be authoritative.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

Open coding models close in on the frontier: DeepSeek V4 Pro scores 80.6% on SWE-bench, Qwen3.8-Max claims 16-day runAug 3, 2026 · Code

Also known as

SWE-bench · SWE bench · SWEbench · SWE-bench Verified · SWE-bench Lite · SWE-bench Pro · Software Engineering Benchmark · software development benchmark

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.