What is being scored

The original SWE-bench paper frames a concrete challenge: given an issue description and a repository, produce a patch for a real software issue. The benchmark's test harness decides whether the patch passes specified tests. OpenAI's August 2024 Verified release described a human-screened 500-task subset intended to reduce underspecified or incorrectly judged tasks. A pass rate therefore describes performance on a particular dataset, scaffold, model configuration, and evaluation procedure. It is not a general probability that an agent will complete the next issue in an arbitrary private repository.

The evaluator can be wrong

In February 2026, OpenAI published an audit arguing that SWE-bench Verified had become unreliable for measuring frontier progress, citing tests that reject valid patches and exposure of benchmark material during training. In July 2026, OpenAI also reported task-quality problems in SWE-bench Pro. These are primary, evaluator-authored critiques and should be read with their own methods and interests in view. They do not prove that every benchmark result is worthless. They do show that a changing score can reflect properties of the test as well as properties of the agent.

Compare complete runs

Ask whether submissions used the same repository snapshot, time allowance, number of attempts, tools, and hidden tests. A model with a search tool and repeated repair loop is not the same system as a model emitting a single patch. Check whether the reported number is pass-at-one, best-of-many, or a selected public run. Note task languages and issue types. Then inspect failure categories: wrong diagnosis, patch rejected by tests, environment failure, or invalid grader. A useful benchmark card makes these denominators visible before naming a winner.

Connect the score to the decision

For procurement or workflow design, construct a small repository-specific evaluation with representative bugs, acceptance criteria, and independent review. Record review minutes and regressions, not only test passes. Preserve the original benchmark result as external evidence and the local trial as separate evidence. This editorial recommendation follows from the benchmark's narrow measurement target and the published audits; it is not a claim that this site has run its own agent comparison. A score should raise a question about task fit, not close the decision by itself. Document the comparison before selecting a tool.

What to carry into the work

  • Record dataset version, scaffold, attempts, and time budget.
  • Inspect task and grader quality audits.
  • Separate public benchmark scores from local evidence.
  • Review accepted patches for behavior beyond tests.
Evidence boundary: this is a sourced editorial guide, not a hands-on product evaluation. Recommendations are our engineering analysis. Product documentation describes intended behavior, not independent proof of reliability.

Sources & dates

  1. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ↗SWE-bench researchers · Undated source · Checked 16 Sept 2026
  2. Introducing SWE-bench Verified ↗OpenAI · 13 Aug 2024 · Checked 16 Sept 2026
  3. Why SWE-bench Verified no longer measures frontier coding capabilities ↗OpenAI · 23 Feb 2026 · Checked 16 Sept 2026
  4. Separating signal from noise in coding evaluations ↗OpenAI · 8 Jul 2026 · Checked 16 Sept 2026

Unknown source dates stay undated. Preparation is not publication; no historical byline or interview is implied.