# SWT-bench Verified leaderboard

external benchmark

> Every figure below is reproduced as its upstream source published it: nothing is modelled, estimated, interpolated or converted. `Unavailable` means no source published the value — it is never a zero. Each value carries its evidence state and the date it was observed.

Page: https://modelchoice.dev/leaderboards/benchmarks/external-swt-bench-verified

SWT-bench Verified: writing tests that reproduce real GitHub issues. Every model the catalog carries a published SWT-bench Verified value for, ranked by that value.

## Benchmark record

| Field | Value |
| --- | --- |
| Category | external |
| Measure | Agent or model gets the issue and the pre-fix repository and adds tests; tests run before and after the reference fix |
| Tasks | 433 human-verified GitHub issues from SWE-bench Verified (Python repositories); SWT-bench Lite has 276 issues from SWE-bench Lite |
| Difficulty | Reproducing a bug from an issue report as a failing test inside a real project's test suite |
| Published by | [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://arxiv.org/abs/2406.12952) |

A test generation benchmark from LogicStar AI and ETH Zurich's SRI Lab. For each of 433 real GitHub issues from SWE-bench Verified, an agent writes a test that reproduces the reported bug: the test has to fail on the original code and pass once the issue's fix is applied. The score is the share of issues reproduced this way.

## SWT-bench Verified ranking

No model in the catalog has a published SWT-bench Verified score. The benchmark is defined by SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents, but the catalog carries no value for it yet. An absent value is shown as absent here rather than as a zero.

Rows are ordered by the value the source published, highest first. The source does not state whether a higher value is the better result, so this document does not either: for a benchmark that measures a rate of failure, read the table from the bottom.
