OpenAI has identified significant quality issues in SWE-Bench Pro, one of the most prominent coding benchmarks used to evaluate AI models' software development capabilities.

The company's detailed audit found that approximately 30% of the 731 tasks in the benchmark's public split contain breaking issues that could misrepresent model performance and affect safety assessments.

SWE-Bench Pro was designed as an improvement over SWE-bench Verified, which OpenAI previously abandoned due to fundamental design flaws. The benchmark tests models on implementing solutions that pass new feature tests without breaking existing functionality.

Four categories of problems identified

OpenAI's analysis revealed four primary failure modes affecting the dataset. Overly strict tests enforce specific implementation details not specified in prompts, invalidating functionally correct solutions. Underspecified prompts omit requirements that hidden tests enforce but aren't reasonably inferable from the task description.

Low-coverage tests inadequately check requested features, allowing incomplete fixes to pass evaluation. Misleading prompts direct models toward incorrect behaviour or contradict test requirements.

The company's datapoint analysis pipeline flagged 200 tasks (27.4%) as broken, while human reviewers identified issues in 249 tasks (34.1%). Five experienced software engineers independently reviewed each flagged task, with disagreements escalated for further investigation.

Methodology combines AI agents and human expertise

OpenAI developed a quality assurance pipeline using investigator agents with access to task repositories and environments. These agents could run tests, inspect files, and investigate common failure modes before human researchers made final judgements.

Human reviewers proved more likely than AI agents to identify broken tasks and often found multiple overlapping issues per task. The largest discrepancy emerged in low-coverage test identification, where humans flagged 9.4% of tasks compared to 4.1% from the agent pipeline.

Frontier models showed dramatic improvement on SWE-Bench Pro over eight months, with pass rates rising from 23.3% to 80.3%. However, OpenAI's findings suggest these gains may be less meaningful than previously understood.

The research highlights ongoing challenges in creating fair, hard benchmarks for AI evaluation and demonstrates the growing utility of AI agents for scalable data quality assessment.