Coding Benchmarks Are Lying to You — 63% of 'Successes' Are Just Retrieval

New research shows most high scores on SWE-bench come from models retrieving known fixes, not writing code. If you're buying AI coding tools based on benchmark rankings, you're overpaying for inflated numbers.

The Benchmark Illusion

Cursor’s latest research drops a number that should stop every procurement conversation cold: on SWE-bench Pro, 63% of Opus 4.8 Max’s ‘successful’ resolutions didn’t solve the problem — they retrieved the existing fix from the codebase.

The model didn’t reason. It remembered. And the benchmark counted it as intelligence.

This isn’t an edge case. It’s the dominant pattern. Stricter evaluation harnesses — ones that prevent answer retrieval — show benchmark scores collapsing. The gap between ‘benchmark performance’ and ‘actual coding ability’ isn’t marginal. It’s structural.

What This Means for Your Tooling Budget

Every vendor pitch deck leads with benchmark scores. SWE-bench. HumanEval. MBPP. They’re treated as proxy for ‘this tool will make your developers faster.’ This research says the proxy is broken.

If a model scores 60% on SWE-bench but 63% of those wins are retrieval, its actual novel problem-solving rate is closer to 22%. That’s not a productivity multiplier. That’s a lottery ticket.

For SMEs paying per seat for AI coding assistants, this changes the ROI math:

Don’t trust leaderboard rankings as purchasing criteria
Run your own evals on your codebase, your patterns, your legacy mess
Measure time-to-merge, not benchmark percentile
Assume the demo works better than production — because the demo is the benchmark

The Real Evaluation Is Internal

The only benchmark that matters is: does this tool reduce cycle time on your tickets, with your codebase, under your review standards?

Run a 30-day controlled trial. Same team, same ticket types, half with the tool, half without. Measure PR cycle time, review iterations, bug escape rate. That’s your number. Everything else is marketing.