Gloaguen, Mündler, Müller, Raychev & Vechev — Evaluating AGENTS.md
TL;DR
A rigorous negative result against a practice every agent vendor recommends and most teams have adopted: writing a repository-level context file (AGENTS.md, CLAUDE.md, and their equivalents) to tailor coding agents to a codebase.
Two evaluation settings, chosen to close the obvious objection that LLM-written context files are strawmen:
- Established SWE-bench tasks from popular repositories, with LLM-generated context files
- A novel collection of issues from repositories containing developer-committed context files
Findings:
- Context files do not generally improve task success rates.
- Inference cost rises by over 20% on average.
- This holds across different LLMs, different coding agents, and both LLM-generated and human-written context files.
The decomposition is the actionable part, and it rescues the practice from being simply wrong:
“While instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful.”
So the file is not useless — its most popular section is. Agents do follow explicit instructions. What they do not benefit from is the architectural tour, the module inventory, the “here’s how this codebase is organised” preamble that sits at the top of nearly every AGENTS.md in existence and is exactly what model providers’ templates ask for. That section is pure token cost, paid on every single invocation.
The authors’ conclusion is measured, not dismissive: context files are useful for specifying non-standard coding practices, and “any attempts to improve performance should be rigorously evaluated before deployment.”
The practical prescription — practitioner shorthand calls it a “lean AGENTS.md,” which is not the paper’s phrase: keep the imperatives, delete the description. Say “run make lint before committing” and “never modify files under generated/.” Delete “this repository implements a three-tier architecture with a service layer that…” — the agent can read the code, and it is charging you 20% to be told about it first.
This lands next to Abujadallah et al., who conclude independently from 306 rejected agent PRs that what agents need is hints about the approach and explicit statements of forbidden approaches — both imperative, neither descriptive. Two studies, different methods, same shape of answer.
It belongs beside METR’s RCT as the corpus’s second rigorous negative result against an intervention the field assumed was obviously beneficial — one on assistance, one on context — both surviving robustness checks.
Dynamic-capabilities reading
digital-transforming/improving-digital-maturity— the paper’s real prescription is evaluative discipline: measure the harness intervention before standardising it, rather than adopting what the vendor’s documentation recommends.contextual/internal-barriers— a universally recommended practice that raises cost >20% for no measured gain is a barrier hiding inside a best practice.
Linked entities and concepts
- Concepts: agent-harness, agentic-engineering, agent-development-lifecycle, ai-benchmarks, llm-wiki
- Dangling (single-source mention, deferred): Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, Martin Vechev
Scope and reliability
Abstract only — the models and agents tested, the effect-size confidence intervals, and how “repository overview” was distinguished from “instructions” in the analysis were not read. That last point is the one that would most change the prescription’s precision. Measured on issue-resolution success, which is a narrow outcome: a context file might reduce review burden, improve style conformance, or lower rejection rates without moving SWE-bench pass@1 at all — none of which this design would detect. Strong provenance (ETH Zürich SRI Lab). Worth noting reflexively: this wiki’s own CLAUDE.md is a repository context file, and it is overwhelmingly imperative rather than descriptive — which is the shape the paper’s evidence supports.