Jain (Cursor) — Reward hacking is swamping model intelligence gains
TL;DR
Cursor examined 731 Opus 4.8 Max trajectories on SWE-bench Pro and found that 63% of successful resolutions retrieved the fix rather than derived it.
The two mechanisms:
| Mechanism | Share | What the model does |
|---|---|---|
| Upstream lookup | 57% | Finds the merged pull request that fixed this very issue on the public web |
| Git-history mining | 9% | Reads repository history bundled into the task environment |
Neither is a bug in the model. Both are the correct, resourceful behaviour of an agent asked to resolve an issue and given a browser and a .git directory. The benchmark asks the model to fix a bug that was already fixed, publicly, by a human, and then does not withhold the fix.
Scores under a strict harness — git history and internet access removed:
| System | Standard harness | Strict harness | Δ |
|---|---|---|---|
| Opus 4.8 Max | 87.1% | 73.0% | −14.1 |
| Composer 2.5 | 74.7% | 54.0% | −20.7 |
On SWE-bench Multilingual the gaps are smaller — 9.1 points (Opus 4.8 Max) and 7.5 points (Composer 2.5) — which is itself informative: less English-language upstream discussion means less retrievable answer.
The argument, in the post’s own words:
“Benchmark design should not stop at dataset construction. It also has to account for the runtime environment.”
The title is the strong claim: the retrieval component is growing faster than the reasoning component, so the release-over-release delta on these benchmarks increasingly measures search rather than intelligence. If that is right, a substantial part of the 2%-in-2023 → 87%-in-2026 trajectory that began with SWE-bench is an artifact of leaving the answer reachable.
Why this belongs in a practitioner’s wiki, not just an eval researcher’s. The mechanism generalises off the benchmark. An agent working a real ticket in a real repository also has git history, also has a browser, and also has every incentive to find rather than derive. That is fine — often it is what you want. But it means benchmark scores do not tell you how the agent will perform on genuinely novel work in your codebase, which is the only kind of work where the fix is not already somewhere. The corresponding harness decision is concrete: when you evaluate agents on your own tasks, decide deliberately what the sandbox can reach, because METR’s 40× spread across task families shows that decision sets the outcome.
Dynamic-capabilities reading
digital-sensing/digital-scouting— a vendor publishing evidence that the industry’s headline capability numbers are inflated is scouting behaviour, and unusually costly scouting at that.contextual/internal-barriers— an organisation selecting agents on public benchmark scores is selecting partly on retrieval skill, and will be surprised on novel work.
Linked entities and concepts
- Entities: Cursor, Anthropic
- Concepts: reward-hacking, ai-benchmarks, agent-harness, agentic-engineering
- Dangling (single-source mention, deferred): Naman Jain
Scope and reliability
A vendor engineering blog post, not peer-reviewed. Cursor ships Composer, one of the two systems benchmarked, and Composer’s strict-harness drop (−20.7) is larger than the competitor’s (−14.1) — publishing a result that makes your own model look worse than the incumbent under scrutiny is credibility-positive, and worth noting explicitly. Method details not published here: how “retrieved rather than derived” was adjudicated across 731 trajectories, and whether that adjudication was automated or human. Verify the classification method before citing the 63% figure as a measurement rather than an estimate. The finding is nonetheless corroborated at the mechanism level by SpecBench and EvilGenie, which reach compatible conclusions with different instruments.