Rethinking Agents — Harness is All you Need
Confidence 0.70 · last confirmed 2026-05-08
Two new papers from Stanford and Tsinghua just put hard numbers on something most agent builders have been feeling — the orchestration code wrapping your LLM now drives more performance variation than the model itself. Same model, six-times the gap, depending entirely on what researchers are calling the harness. If you build agents, the lever you should be pulling is almost never the one you’ve been reaching for.
— Prompt Engineering YouTube channel description (sponsor mention and link list omitted for brevity; full text in
raw/videos/rethinking-agents-harness-is-all-you-need.md)
A ~14-minute YouTube explainer on the Prompt Engineering channel, published 4 May 2026 — the wiki’s first video ingest. Synthesises two recent academic papers on harness engineering: Pan et al. (Tsinghua, March 2026) on natural-language harness representation, and a paper from Omar Khattab’s team (DSPy authors) on automatic harness optimization. Headline claim: “the orchestration code wrapping your LLM now drives more performance variation than the model itself — same model, six times the performance gap depending entirely on the wrapper or harness around it.”
The video is the wiki’s first empirical anchor for the harness-as-moat framing. Where Chatterjee 2026 and Kokane 2026 argued rhetorically about harness importance from practitioner experience, this source brings ablation numbers, benchmark deltas, and a transfer-across-models result. The presenter (unnamed in the transcript) explicitly frames harness engineering as a discipline name and ends with a four-question audit checklist for builders.
The presenter is not identified in the transcript; channel attribution (Prompt Engineering) is the strongest provenance available.
TL;DR
- The 6× claim: same foundation model, same prompt, very different success rates depending on harness — Claude / GPT / Gemini behave differently in Claude Code vs Cursor vs Anti-gravity vs OpenCode. Harness, not model, drives the variance.
- Pan et al. (Tsinghua, March 2026) — Natural-Language Harness: rewrite control logic from Python/YAML into structured natural language. 3-layer split: back-end (tools/infrastructure) → runtime charter (universal physics: contracts, state, sub-agent management) → natural-language agent harness (state-specific logic: contracts, roles, state structure, failure modes). On SWE-bench verified with GPT-5 max-reasoning, the full harness used 16.3M tokens / 600+ tool calls / 32 minutes for 74–76%; a stripped variant achieved the same with 1.2M tokens / 51 calls / under 7 min — 14× compute waste. Module-by-module ablation: self-evolution was the only consistently helpful module; verifiers actually hurt (−0.8 on SWE-bench, −8.4 on OS World); multi-candidate search hurt by −5.6.
- Pan et al. — natural-language migration win: took OS-Symphony (a native-code harness for desktop automation) and rewrote its logic in natural language with the same strategy. 30.4% → 47.2% performance, runtime 361 min → 41 min, LLM calls 1,200 → 34. “The representation itself drew the gain.”
- Khattab et al. — Meta-Harness (auto-optimisation): an agentic proposer (Claude Opus 4.6) reads failed execution traces, diagnoses what broke, and rewrites the entire harness. Loop scale: 10M tokens/iter, 400× more feedback than any prior method, 82 files read per round. Removing raw traces drops accuracy 50% → 34%; replacing traces with summaries: 34.9%. “This signal lives in the raw details. If you summarise prior failures and feed those into the model, you can hurt the agent.” Best result: 76.4% on Terminal Bench 2, the only auto-optimised system in a hand-engineered field; 7.7 points above SOTA on a 215-class text classification using 4× fewer tokens, with Haiku ranked above Opus.
- Transferable harness: a harness optimised on one model improved five other models without re-optimisation. “The reusable asset is not the model — it’s the harness.”
- Anthropic’s “subtraction principle” — the core practical takeaway: every harness component encodes an assumption about what the model cannot do alone, and those assumptions expire when models improve. Manus rewrote their harness 5× in 6 months; Warel removed 80% of agent tools and got better results. Mature harness work looks less like adding structure and more like pruning it.
- The audit checklist for builders (when an agent underperforms — don’t switch the model first): (1) what’s in the context window that doesn’t need to be there? (2) which tools does the agent rarely use well? (3) are verification or search loops hurting performance (per the Tsinghua ablations)? (4) is your control logic in code or in language?
What was actually ingested
Full transcript, ~14:10 minutes / ~580 lines. The transcript carries one mid-video sponsor break (Data Impulse residential proxies, ~6:29–8:01) which is ignorable for the wiki and is excluded from the ingest. The ~12-second segment immediately after the sponsor (the “Meta Harness Auto Optimization” intro line that begins “It’s structured retrieval memory orchestration topology”) is the start of the Khattab section and is included.
Transcript timestamps:
- 0:00–1:13 — headline 6× claim, three things the video covers.
- 1:13–2:44 — what is a harness (CPU/RAM/disk/OS analogy).
- 2:44–4:03 — Pan et al. paper, 3-layer NL harness structure.
- 4:03–6:29 — Pan et al. ablations + OS-Symphony NL migration win.
- 6:29–8:01 — sponsor break (skipped).
- 8:01–10:02 — Khattab et al. Meta-Harness auto-optimization loop, scale, transferability.
- 10:02–11:32 — same-model 6× variance demonstrated across harnesses (Claude Code, Cursor, Anti-gravity, OpenCode).
- 11:32–13:09 — Anthropic’s subtraction principle; Manus and Warel as evidence.
- 13:09–14:10 — audit checklist for builders + close.
What is a harness — the operating-system analogy
Before either paper is introduced, the video offers a vivid mnemonic for what a harness is and is not:
“The raw LLM is the CPU — powerful but inert. No memory, no storage, no IO. Your context window acts as RAM — fast but limited. External databases serve as disk. Tool integrations are device drivers. And the harness is the operating system, deciding what the CPU sees and when.”
| OS-stack component | Agent-stack equivalent |
|---|---|
| CPU — powerful but inert | The raw LLM |
| RAM — fast but limited | The context window |
| Disk | External databases / state stores |
| Device drivers | Tool integrations |
| Operating system — schedules what the CPU sees | The harness |
The mnemonic does two things at once:
- It inverts the headline of most AI discourse: the model is the CPU, not the system. A CPU does nothing useful without an OS scheduling its inputs. By analogy, an LLM does nothing useful in production without a harness deciding what enters its context window and when.
- It anchors the harness’s job description: an OS does memory management, scheduling, IO mediation, process isolation, and exception handling. An agent harness does context management, agentic-loop control, tool execution, sandboxing/permissions, and retry/error handling. The mapping is structural, not metaphorical.
The presenter explicitly frames this as the conceptual scaffolding for the harness components covered in the channel’s previous video (the “nine harness components”). For the wiki, the analogy is the cleanest single-sentence answer to “what does a harness do?” — preserved here for cross-source citation.
The two underlying papers (referenced second-hand)
The video synthesises two academic papers we have not separately ingested. Both are dangling source candidates — primary-source ingest would substantially strengthen the empirical anchor on the agent-harness concept.
- Pan et al., Tsinghua University, March 2026. Natural-language harness representation; the 3-layer back-end / runtime-charter / NL-harness split; the SWE-bench ablation results; the OS-Symphony migration. Citation in the video is informal — first author surname only.
- Khattab et al. (DSPy team), “Meta-Harness” / agentic-proposer auto-optimization. Terminal Bench 2 76.4%; transferable across 5 models; raw-traces-irreplaceable finding. No date stated; presenter says “released by Omar Khattab who built DSP”.
The video is therefore third-hand testimony for the underlying paper claims — useful as a synthesised summary but not an authoritative substitute. Treat the specific numbers as second-hand until a primary-source ingest confirms them.
Pan et al. — natural-language harness representation
The paper’s premise: agent control logic is conventionally written in Python or YAML. What if it were written in structured natural language instead?
The proposed 3-layer architecture:
| Layer | Content | Function |
|---|---|---|
| Top | Natural-language agent harness | State-specific logic: contracts, roles, state structure, failure modes |
| Middle | Runtime charter | ”Universal physics” — how contracts bind, how state persists, how sub-agents are managed |
| Bottom | Back-end | Actual infrastructure and tools |
Why the split matters: it gives harness engineering controlled experimentation. Swap the harness while fixing the runtime — clean ablation, harness design tested in isolation.
The ablation result
On SWE-bench verified with GPT-5 at maximum reasoning:
- Full harness: 74–76% pass rate, 16.3M prompt tokens/sample, 600+ tool calls, 32+ min runtime.
- Stripped variant: 1.2M tokens, 51 calls, < 7 min — same destination, 14× the compute saved.
Module-by-module ablation:
- Self-evolution — the only module consistently helpful when added.
- Verifiers — actively hurt: −0.8 on SWE-bench verified, −8.4 on OS World.
- Multi-candidate search — hurt by −5.6.
“More structure is not always better, which is kind of a surprising result, but if you think about it, it kind of makes sense.”
This empirically grounds Chatterjee’s Constraints layer warning that post-tool hooks done badly are worse than no hooks, and Anthropic’s subtraction principle (below).
The natural-language migration win
OS-Symphony, a native-code harness for desktop automation, was migrated to a natural-language representation using the same strategy. Result:
- Performance: 30.4% → 47.2% (+16.8 points).
- Runtime: 361 min → 41 min (~9× faster).
- LLM calls: 1,200 → 34 (~35× fewer).
“The representation itself drew the gain.”
The headline interpretation: the substrate in which control logic is expressed (code vs structured natural language) is itself a load-bearing harness-design decision — not just a packaging detail.
Khattab et al. — Meta-Harness (auto-harness optimisation)
If representation matters this much, can the right harness be discovered automatically?
The architecture (per the video):
- An agentic proposer (using Claude Opus 4.6) reads failed execution traces, diagnoses what broke, and rewrites the entire harness.
- The loop runs at scale: 10M tokens per iteration, 400× more feedback than any prior method, ~82 files read per round.
- Final scores plus raw traces accumulate in a growing file system. The loop repeats.
Raw traces are irreplaceable
A specific empirical finding worth surfacing for builders:
| Trace handling | Accuracy |
|---|---|
| Full raw traces fed to proposer | 50% |
| Traces removed entirely | 34% |
| Traces summarised before being fed | 34.9% |
“This signal lives in the raw details. If you summarise prior failures and just feed those into the model, it could hurt the performance of your agent.”
This is a sharp counterintuitive result — the standard “trim the context, summarise the history” reflex actively destroys the signal needed for self-improvement. It generalises, plausibly, to any harness compounding-loop (Chatterjee’s Compounding layer).
Best results
- 76.4% on Terminal Bench 2 — the only auto-optimised system in a field of hand-engineered entries.
- +7.7 points above state-of-the-art on a 215-class text classification benchmark, using 4× fewer tokens.
- Model rankings inverted: Haiku ranked above Opus on the auto-optimised harness — a smaller model outranking larger ones through harness optimisation alone.
Transferability — the load-bearing finding
“A harness optimised on one model transferred to five other models and improving them all.”
The presenter explicitly draws the implication:
“The reusable asset is not the model, it’s the harness. You can build it once. It works across the model landscape.”
This is the strongest empirical support yet for Chatterjee’s “the model is rented; the harness is owned” framing on agent-harness. Harness work compounds across model upgrades; model-tuning work doesn’t.
Same-model variance across harnesses (the qualitative demo)
The video closes the empirical section with a simple thought-experiment readers can run themselves:
“Take any frontier model — Claude Opus 4.6 or 4.7, GPT-5.4, Gemini, doesn’t matter which. Run the exact same prompt through it inside Claude Code, then Cursor, then Anti-gravity. Keep the model exactly the same — you just change the harness. You’ll see noticeably different results: different reasoning paths, different token spends, different success rates. All of this is driven not by the model but by the harness.”
The instance specifically named: “Opus 4.7 behaves very differently in Claude Code than inside Cursor or other third-party harnesses like OpenCode.”
Anthropic’s “subtraction principle”
The most concentrated practical takeaway. Framed as the answer to “what most builders are getting wrong”:
“Every harness component encodes an assumption about what the model cannot do alone — and those assumptions expire when models improve. When Opus 4.6 stopped needing context resets, Anthropic just dropped them entirely.”
Two corroborating data points named:
- Manus, the agent platform, rewrote their harness 5× in 6 months.
- Warel (mentioned by name; not separately context-set in the transcript) removed 80% of the agent’s tools and got better results.
“Mature harness work looks less like building structure up and more like pruning it down.”
This directly contradicts the late-2024 / early-2025 instinct to give agents more and more tools. Combined with the Pan et al. ablation finding that verifiers actively hurt, the prescription is operational: if the model can now do an aspect that was hand-crafted in the harness before, drop the hand-crafted machinery.
The subtraction principle is the closest the wiki has yet come to a named operational discipline for harness pruning — distinct from but compatible with Chatterjee’s additive 4-layer build order. Both can be true: build the four layers; prune within each layer ruthlessly as the model improves.
The audit checklist for builders
“If you are building an agent, you are a harness engineer whether you call yourself one or not. So when you run into something that is not working, here is the order of operations I would suggest. Don’t switch the model first. Audit the harness.”
The four questions:
- What’s in your context window that does not need to be there?
- Which tools does the agent rarely use well? (or: rarely use at all)
- Are you adding verification or search loops that might be hurting performance (per the Tsinghua ablation)?
- Is your control logic written in code or in language? (per the OS-Symphony 17-point migration win)
“It is no longer a question of which model to pick. It is a question of which structure to remove.”
Cross-source positioning
This video sits as the empirical anchor beneath the rhetorical/architectural anchors already in the wiki:
| Source | Vantage | What it adds to agent-harness |
|---|---|---|
| 2026-05-07-anthropic-managed-agents-decoupling-brain-hands | Vendor productisation (8 Apr) | Brain / hands / session decomposition; managed-runtime as a service |
| 2026-04-22-cheung-ippolito-secchi-google-agents-cli | Vendor productisation (22 Apr) | 7 specialised skills + 9-stage ADLC; CLI-as-harness-toolkit |
| 2026-05-07-kokane-agent-harness-vs-systems-design | Sceptical practitioner essay | ”90% systems engineering rebranded; 10% genuinely novel” framing + 4-layer stack diagram |
| 2026-05-07-chatterjee-anatomy-of-agent-harness | PM-facing practitioner essay | 4-layer build order: Context / Constraints / Contracts / Compounding; “model rented, harness owned” framing |
| 2026-05-07-kiron-schrage-compound-benefits | MIT SMR column | Verification → evaluation → learning capture flywheel at the org level |
| This video (4 May) | Empirical synthesis | 6× variance ablation data, 14× compute waste, transferable-harness finding, raw-traces-irreplaceable, subtraction-principle as named discipline |
The most important new contribution: transferability. Until now, the wiki has held the “model is rented; harness is owned” framing as a rhetorical claim. Khattab’s transferable-harness finding makes it an empirical one.
The second important new contribution: the subtraction principle as a named operational counter-prescription to the additive build-everything instinct. It pairs naturally with Chatterjee’s additive 4-layer build order and Kokane’s “if you’ve shipped real backend systems, you’re already 80% of the way there.”
Linked entities and concepts
Existing pages (touched or referenced):
- agent-harness — primary concept; this source becomes its 6th citation and its first empirical anchor.
- ai-agents — the broader concept the harness sits inside.
- foundation-models — the substrate the harness wraps; “transferable harness” reinforces the rented-model framing.
- 2026-05-07-chatterjee-anatomy-of-agent-harness — the rhetorical anchor whose claims this video empirically supports.
- 2026-05-07-kokane-agent-harness-vs-systems-design — the sceptical vantage; the subtraction principle is operationally what Kokane’s “10% genuinely novel” looks like in practice.
- 2026-05-07-anthropic-managed-agents-decoupling-brain-hands — Anthropic’s broader operational stance; the subtraction principle is named here as Anthropic’s.
- 2026-04-22-cheung-ippolito-secchi-google-agents-cli — referenced as the canonical “Claude Code as a harness” example used in the same-model-variance demo.
Dangling (single-source mention, deferred per CLAUDE.md author-entity promotion rule):
- Prompt Engineering (YouTube channel; placed in
author:per video convention). - Pan et al. — Tsinghua, March 2026 — first author surname only; primary paper not ingested.
- Omar Khattab — DSPy creator; first mention by name in the wiki.
- Manus (agent platform).
- Warel (agent platform / company; spelling unverified — transcript-derived).
- DSPy — referenced in passing in Kokane 2026 as a Layer-2 framework, but no entity page yet; defer until a third source warrants it.
Source-quality flag
- Strengths: explicit empirical claims with named benchmarks (SWE-bench verified, OS World, OS Symphony, Terminal Bench 2); concrete 14× / 35× / +16.8-point deltas; names two papers with first-author identification; presenter is comfortable with the technical material and ends with an operational checklist.
- Caveats: opinion-format video; third-hand on the underlying papers (presenter is summarising, with specific numbers possibly mis-rounded — “GPD54” likely means “GPT-5.4”, “DSP” likely means DSPy); presenter is unnamed; the “Warel” reference is unverified spelling; no link to the underlying papers in the transcript itself; one paid sponsor segment (Data Impulse) excluded from ingest.
- Per CLAUDE.md §Lifecycle: confidence baseline 0.70 (single supporting source for the new empirical claims). Capped at 0.75 because video is opinion-format and the cited numbers are second-hand. The video’s primary contribution to the wiki is the empirical anchor on transferability and the subtraction principle as a named discipline, both of which are durable claims even if specific numbers need primary-source verification.
Open questions
- Primary-source ingest of Pan et al. (Tsinghua, March 2026). Would substantiate the SWE-bench ablation numbers and the OS-Symphony migration deltas. Search target: “Pan natural-language agent harness Tsinghua March 2026” + arXiv.
- Primary-source ingest of Khattab et al. (Meta-Harness). Would substantiate the Terminal Bench 2 76.4%, the transferability claim across five models, and the raw-traces-irreplaceable finding. Search target: Omar Khattab + Stanford / DSPy team + auto-harness optimisation.
- Does the “raw traces > summaries” finding generalise? The video presents it as a Khattab-team result on auto-optimisation; if it holds for any harness compounding loop, it has direct implications for Chatterjee’s Compounding layer. Worth tracking.
- Manus and Warel as candidate entity pages. Both are named as agent platforms with relevant operational claims. Promote on second-source mention.
- Presenter identity. Prompt Engineering is the channel; the host’s name is not stated in the transcript. Backfill if surfaced.
- Verifiers hurt — why? Pan et al. found verifiers degrade SWE-bench by 0.8 and OS World by 8.4. The video does not explain the mechanism. Hypothesis: verifiers add latency without catching enough real failures to justify the noise they inject; or verifier-driven retries dilute the agent’s confidence calibration. Worth tracking against future ablation work.