An open, reproducible academic benchmark with no placement to buy — though it thanks OpenAI and Anthropic for support, and critics, including OpenAI, say training-data contamination has eroded what its top scores actually prove.
What it's really for An academic coding-agent benchmark; agents are graded automatically against real test suites.
What our grade covers The grade on this page is about its score for resolving real GitHub issues, not everything the site does.
High Scoring Confidence Checked against primary sources. We are confident in the facts and the grade here.
No one pays to be ranked and there is no placement to buy — labs submit results voluntarily, submissions are free and public via the open sb-cli tool, and models are scored by the same open test harness. But the leaderboard page thanks "Open Philanthropy, AWS, Modal, Andreessen Horowitz, OpenAI, and Anthropic" for their support, so two of the labs whose models it ranks are among the institutions backing it, and SWE-bench Verified was built in collaboration with OpenAI.
Source →- Operating since
- 2023 (3 years) · source
- What it costs you
- Free to read The reviews are free to read.
- How they make money
- It doesn't sell anything: it's a free, MIT-licensed academic research benchmark out of Princeton with no ads and no paid placement, funded by institutional support it names on the leaderboard page — Open Philanthropy, AWS, Modal, Andreessen Horowitz, OpenAI and Anthropic.
- What they do
- It ranks AI coding agents by how many real GitHub issues they actually resolve, scored automatically against the projects' own test suites via an open, containerized evaluation harness.
- What to watch for
- A high SWE-bench score isn't proof of real coding skill: researchers have documented "solution leakage" and, by OpenAI's own February 2026 disclosure, frontier models had seen the test tasks in training, so rankings can reward memorization.
- Composite score
- 4.00 / 5.00 → grade A-
- Last verified
- July 11, 2026 — when a human last checked this entry's facts, links, and grade against the live site.
- Citations checked
- September 1, 2026 — our automated check re-fetched all 8 sources cited on this page. All resolved. 1 block automated visitors and are checked by hand. This is a link check, not a re-grade — the “Last verified” date above is the one that covers the facts and the grade.
- Site confirmed live
- September 1, 2026 — our monthly automated check reached SWE-bench and got a normal response.
How the grade was reached
Does the site take money from the very entities it ranks? Pay-for-placement, vendor-funded data, and affiliate commissions all pull this down. The less the ranking can be bought, the higher the score.
What is the ranking actually built on? Hands-on testing scores highest, then verified first-hand reviews, then opinion or popularity surveys and self-reported figures, then pay-to-rank, which scores lowest.
Is the methodology published, specific, and reproducible? Can a reader see how a given rank was reached, or is it a black box?
Are commercial relationships, sponsorships, and affiliate arrangements disclosed clearly and near the rankings themselves, rather than buried?
How hard is it to game? Controls against fake reviews, solicited reviews, and vendor gaming raise this; an open box anyone can stuff lowers it.
Evidence
- SWE-bench is a benchmark for evaluating large language models on real-world software issues collected from GitHub; given a codebase and an issue, a model generates a patch that is then verified against the repository's own tests. The evaluation harness is open source, MIT-licensed, and uses a fully containerized Docker setup for reproducible evaluations, with leaderboard submissions run through the open sb-cli tool. Source: SWE-bench GitHub repository (SWE-bench/SWE-bench) →
- SWE-bench Verified is a 500-instance human-filtered subset created in collaboration with OpenAI, where human annotators reviewed each instance to ensure problem descriptions are clear, test patches are correct, and tasks are solvable. OpenAI's collaboration is disclosed openly on the benchmark's own Verified page. Source: SWE-bench Verified (swebench.com) →
- OpenAI stopped evaluating models against SWE-bench Verified on February 23, 2026, after an audit found 59.4% of failed test cases were flawed and that every frontier model (GPT-5.2, Claude Opus 4.5, Gemini 3) showed training-data contamination — models trained on post-June-2024 GitHub data had seen the 500 Verified tasks, including solutions. Source: OpenAI: Why we no longer evaluate SWE-bench Verified →
- The leaderboard page carries an acknowledgements line: "We thank the following institutions for their generous support: Open Philanthropy, AWS, Modal, Andreessen Horowitz, OpenAI, and Anthropic." Two of the six, OpenAI and Anthropic, build models that appear on the leaderboard. The disclosure sits on the same page as the ranking, but names only the supporters, not what the support consisted of. Source: SWE-bench Leaderboards — Acknowledgements →
- The default SWE-bench Verified view is a bash-only comparison in which every language model is run through the same minimal mini-SWE-agent ReAct loop — "No tools, no special scaffold structure" — rather than through each vendor's own agent scaffold, and entries are flagged where the run was "performed or directly checked by the SWE-bench team." Submission remains free and open to anyone through the public SWE-bench/experiments repository and the sb-cli tool; there is no paid or expedited submission tier. Source: SWE-bench Verified + Submit pages →