Abujadallah, Arabat & Sayagh — Understanding the Rejection of Fixes Generated by Agentic Pull Requests
TL;DR
46.41% of the fixes proposed by Copilot, Devin, Cursor and Claude in the AIDev dataset are rejected.
That is the number to hold against every practitioner throughput claim in this ingest. The authors’ framing is deliberately economic: this is “a significant amount of wasted resources that require human reviews, verifications, and running tests and validations for fixes that are merely discarded.” Agent output is not free at the point of rejection — it consumes reviewer attention, CI capacity, and tokens, all spent on work thrown away.
Method: qualitative study of a representative sample of 306 non-merged pull requests, followed by quantitative analysis of the reasons. Output: 14 rejection reasons in 4 high-level categories.
| Category | Examples |
|---|---|
| Incorrect implementation | Incomplete fix; wrong approach taken |
| CI/test failure | Doesn’t pass the pipeline; breaks tests |
| Agent unable to implement | No code generated; sessions lost |
| Low priority | The fix is fine; nobody wanted it |
The fourth category is the interesting one. “Low priority” rejections are not agent failures at all — they are targeting failures. The agent did the work correctly and nobody needed it. That cost is invisible in any metric that counts PRs authored, and it is the population-scale version of Carson’s own anti-thesis — “I don’t get multiples of quality off of multiples of output” — which he reached by intuition and rations with a folder hierarchy and a paper priority list.
The authors’ prescriptions map onto harness design directly:
- Hint the approach — tell the agent how to fix it, not just what to fix
- State the forbidden approaches — constrain the solution space explicitly
- Instruct the agent on validation — how to get through CI without introducing a breaking change
- Prioritise properly — so correct fixes are not discarded as unwanted
Prescriptions 1 and 2 are worth reading next to Gloaguen et al., whose finding is that repository context files help only where they carry explicit instructions and are useless where they carry repository overviews. Both point the same way: specific procedural constraints earn their tokens; general context does not. Prescription 4 is not a harness change at all — it is management, and it is the one nothing in the tooling can supply.
The merged tail is characterised by the companion MSR 2026 study, LGTM, on the same dataset: auto-merged agentic PRs are smaller and more focused, and repositories tend to auto-merge all agentic PRs or none. Read together, the population is bimodal — nearly half of what reaches human review is discarded, and a large share of repositories have stopped reviewing at all.
Dynamic-capabilities reading
contextual/internal-barriers— a 46% rejection rate is the single hardest barrier figure in this ingest, and it sits between agent output and delivered value.digital-transforming/redesigning-internal-structures— the prescriptions are structural: task specification, constraint articulation, validation instruction and prioritisation are all how work is handed over, not how it is executed.
Linked entities and concepts
- Entities: GitHub, Devin, Cursor, Cognition, Anthropic, OpenAI
- Concepts: agentic-pull-requests, agent-fleet-management, multi-agent-failure-modes, agent-oversight-and-delegation, agentic-engineering
- Dangling (single-source mention, deferred): Mahmoud Abujadallah, Ali Arabat, Mohammed Sayagh
Scope and reliability
Abstract only — the 14 individual reasons are not enumerated here, only the four categories. Anyone building a rejection taxonomy from this must read the paper. Two caveats on the headline figure. First, 46.41% is a rejection rate among PRs that were opened, not a defect rate — the “low priority” category alone shows that some rejections reflect correct work nobody wanted. Second, it is computed over public GitHub with five agents and self-identified attribution, inheriting all of AIDev’s coverage limits. The qualitative sample (306) is small relative to the population but appropriate for hand coding, and its representativeness is claimed rather than demonstrated here.