SWE-Bench: Benchmarking AI Coding Agents

How SWE-Bench Works

The core idea behind SWE-Bench is simple but powerful: take a real software issue from a real project, give the agent the issue description and the repository state at the time the issue was filed, and see if the agent can produce a patch that resolves it. The patch is verified by running the project's test suite, specifically the tests that were added or modified in the human-written fix. If the agent's patch passes those tests, the issue is counted as resolved.

Each SWE-Bench task instance contains four components. The issue description is the natural language text from the original GitHub issue, including whatever context the reporter provided. The repository snapshot is the complete codebase at the commit immediately before the human fix was applied. The gold patch is the human-written solution that actually resolved the issue. The test cases are the tests from the gold patch that verify the fix works correctly.

The evaluation process is fully automated. The agent receives the issue description and access to the repository, then produces a patch in unified diff format. The benchmark infrastructure applies the patch to the repository, runs the relevant test suite, and checks whether the tests pass. There is no human judgment involved in scoring, which makes results reproducible and comparable across systems.

The repositories included in SWE-Bench span a range of Python projects with different architectures, coding styles, and domains. Django represents a large, mature web framework. Flask is a smaller, more focused web framework. scikit-learn covers machine learning. sympy handles symbolic mathematics. matplotlib is for data visualization. requests handles HTTP networking. These diverse codebases ensure that strong performance requires genuinely general software engineering capability rather than familiarity with a single project or coding style.

SWE-Bench Variants

The original SWE-Bench dataset contains 2,294 task instances collected from the twelve repositories. Running the full benchmark is computationally expensive, requiring each agent to process thousands of large codebases and generate patches for each. This led to the creation of smaller, more focused variants.

SWE-Bench Lite contains 300 task instances selected to be representative of the full dataset while being practical to run in a reasonable time. The selection criteria include diversity of repositories, difficulty levels, and issue types. Lite provides a quick approximation of full SWE-Bench performance, useful for rapid iteration during agent development.

SWE-Bench Verified is the variant that has become the standard reference for leaderboard comparisons. It contains 500 instances that human annotators have reviewed and confirmed are solvable given only the information in the issue description. This review process eliminated cases where the original issue description was too vague, referred to information not available in the repository, or required knowledge that could not reasonably be inferred from the codebase. By filtering out unfairly difficult cases, Verified provides a cleaner measurement of genuine agent capability.

The distinction between these variants matters when comparing results. A system reporting 40% on the full SWE-Bench might score 50% on Verified, because Verified removes the unsolvable cases that drag down scores on the full dataset. Always check which variant a reported score refers to before comparing numbers across different systems.

What SWE-Bench Actually Tests

Resolving a SWE-Bench task requires a chain of skills that mirrors real software engineering work. The agent must read and understand the issue description, often written in informal language with incomplete information. It must navigate a large codebase to find the relevant files and functions. It must understand how the existing code works, identify the root cause of the reported problem, design a fix that addresses the root cause without introducing new issues, and generate a syntactically correct patch that passes the project's test suite.

Repository navigation is often the hardest part. The repositories in SWE-Bench contain thousands of files organized in complex directory structures. The issue description rarely specifies which files need to change. The agent must use the issue description's clues, keywords, stack traces, and references to specific features to locate the relevant code, a skill that requires understanding both the project's architecture and the domain it operates in.

The difficulty distribution across SWE-Bench tasks is wide. Some issues are straightforward one-line fixes where the issue description essentially describes the solution. Others are complex multi-file changes that require deep understanding of the project's internal design patterns, data flow, and edge cases. This distribution means that even a system with a low overall solve rate might be reliably solving the simpler issues, which constitute a real and valuable category of engineering work.

What SWE-Bench does not test is equally important to understand. It does not test the ability to write code from scratch for new projects. It does not test code review, documentation, testing, refactoring, or architectural design. It does not test collaboration skills, communication with stakeholders, or the ability to handle ambiguous requirements that require clarification. It tests one specific, important skill: reading a bug report and producing a fix for an existing codebase.

How Leading Systems Approach SWE-Bench

The architectures that perform best on SWE-Bench share several common patterns that reflect effective strategies for automated software engineering. Understanding these patterns provides insight into what makes coding agents effective beyond this specific benchmark.

Top-performing systems use multi-step workflows rather than single-shot generation. Instead of reading the issue and immediately generating a patch, they follow a structured process: analyze the issue, search the repository for relevant code, form hypotheses about the root cause, test those hypotheses by examining specific files and functions, design a fix, generate the patch, and verify the patch by running available tests or analyzing the change for logical correctness.

Repository navigation strategies vary but all successful systems have explicit mechanisms for finding relevant code. Some use embedding-based search to locate files semantically related to the issue description. Others use the project's file structure, import graphs, and naming conventions to narrow the search space. The most effective systems combine multiple search strategies and refine their focus iteratively based on what they find at each step.

Iterative refinement is another common pattern. Rather than committing to a single fix attempt, leading systems generate a candidate patch, evaluate it against available information, and revise if the evaluation reveals problems. Some systems run the project's test suite as part of their evaluation loop, catching errors before final submission. Others use the language model itself to review the patch for logical correctness and edge cases.

Multi-agent architectures have shown particular strength on SWE-Bench. These systems assign different roles to specialized agents: one for issue analysis, one for repository exploration, one for patch generation, and one for review. The separation of concerns allows each agent to focus on a specific skill, and the review agent provides a quality check that catches errors the generation agent misses.

Interpreting SWE-Bench Scores

A SWE-Bench Verified score of 50% does not mean the system can handle half of your team's bug reports. The relationship between benchmark scores and production performance is real but indirect, and understanding the translation requires accounting for several factors.

The task distribution in SWE-Bench skews toward well-documented issues in well-maintained, well-tested open-source Python projects. Production bug reports are often vaguer, span more languages and frameworks, and exist in codebases with less comprehensive test coverage. The clean evaluation environment of the benchmark does not include the noisy reality of production development: incomplete documentation, outdated dependencies, custom build systems, and organizational coding conventions.

The scoring is binary: each issue is either resolved or not. In practice, a partially correct fix that identifies the right file and function but gets a detail wrong still demonstrates valuable capability. An agent that consistently gets 80% of the way to a correct fix is extremely useful as a tool that accelerates human developers, even if its SWE-Bench score only counts fully correct solutions.

The benchmark also does not measure cost or time. A system that achieves 50% by spending $5 and ten minutes per task is very different from one that achieves 52% by spending $50 and two hours per task. For production use, the efficiency of the solution matters as much as the accuracy, and SWE-Bench scores alone do not capture this dimension.

Despite these caveats, SWE-Bench remains the best available proxy for coding agent capability. Systems that score well on SWE-Bench consistently outperform lower-scoring systems on real engineering tasks in side-by-side comparisons. The benchmark's grounding in real issues from real projects gives it a validity that synthetic benchmarks cannot match. Use the scores as a strong signal, but not as the only signal, when evaluating coding agents for your team.