Ryan Lopopolo — Harness Engineering: How to Build Software When Humans Steer and Agents Execute (AI Native DevCon)

Reasoning models and coding agents have changed what is possible in software development, but most teams are still using workflows built for a world where humans write nearly all the code directly. That mismatch is where much of the disappointment around AI coding comes from. This talk introduces Harness Engineering: an agent-first approach to building software where the key engineering task shifts from manually producing every implementation detail to designing the harnesses that let agents do useful work safely, repeatedly, and at high quality. Ryan shares practical patterns for making AI coding actually work in production: how to scope autonomy, shape context, design eval and verification loops, structure tool access, and build feedback systems … and why “just give the model the repo” is not a serious deployment strategy.

— Channel description, AI Native Dev (AI Native DevCon talk)

A ~30-minute conference talk at AI Native DevCon by Ryan Lopopolo (Member of Technical Staff, OpenAI) — “having kind of invented the term” harness engineering. This is the conference-talk companion to his OpenAI Codex blog case study (the wiki’s first vendor-side production case study of harness engineering): where the blog showed the artifact (one repo, five months, ~1M LOC, 0 hand-written lines), this talk gives the named definition and the operating discipline. Its load-bearing contribution is the cleanest first-person statement of harness engineering by its coiner, plus a concrete three-phase context-delivery model and a counter-intuitive shift-right prescription.

TL;DR

1. Origin story. ~June last year, with the earliest reasoning models (o3) and earliest Codex CLI, Lopopolo had “an insane idea” to get the tool to do his job — asked the agent to read his Slack alerts channel and triage a page; it couldn’t. He shifted into “presenting myself as a tool to the model in order to empower it,” accumulating a snowball of effective use by giving it more powerful tools + more context about what the job means.

2. December “singularity-level” coding; retool every point release. With GPT 5.2 + Opus 4.5 (December) code production became “something these tools do insanely well” — a once-a-decade disruption (last: the cloud). The pace forces constant re-evaluation of priors: “if you’re not in the habit of completely retooling your stack with every point release of the model, you are missing out.”

3. The core axioms of software are upended. Models now do significant parts of the whole SWE lifecycle — not just writing code, but debugging, triaging, responding to customers, planning, scheduling. Code production was the expensive/slow constraint that dominated headcount; now “give a prompt to a coding agent and get a PR or six.” The goal for all engineers/EMs/PMs/designers is to unblock the execution team — the coding agents — so they can make your vision real.

4. The three remaining foundational limits. In a human+agent team, three constraints persist:

• Human time is the fundamentally scarce resource. He maxes at ~3 concurrent sessions; higher throughput requires removing his own synchronous attention from the process.
• Human and model attention — “attention must sum to one”; thrashing agents with conflicting/overbearing requirements degrades performance. Retool to be parallel: fork more tasks, accept many more (smaller or larger) PRs to let agents explore.
• The context window — still scarce though growing. GPT-series auto-compaction is “fantastic” (tasks run 6/12/36 hours), but the window is “obliterated and rebuilt” across compactions, so you must structure and continually resurface context.

5. Write it down — make NFRs legible. Agents have no presence in standup, no “durable memory that accumulates context and battle scars.” Articulating the non-functional requirements of good software as text (the thing the LM craves) is a net-new function for a SWE team in 2026 — but it’s not enough to write it down; the text must be pulled into context at the right time without thrashing the agent.

6. The definition. “Harness engineering is making context around what it means to do a good job legible, and then just-in-time surfacing it to the agent over the course of its trajectories in order to steer and refine its output” — so every PR adheres to “the golden thread” of acceptable, high-quality, aligned software. The governing rule: “I never want to give the same review feedback twice” → make every mistake statically impossible going forward.

7. Shift RIGHT, not left (counter-intuitive). Normal DevOps wisdom is shift-left (cheaper earlier); with agents he puts interventions as far right as possible to minimize his own synchronous time. There’s a ladder of durability for any correction: trash-and-reprompt (not durable) → write it down → empower a reviewer agent to judge every diff → encode as statically-verifiable lints / guardrails / tests further left. Drive corrections down the ladder over time.

8. Prune latent space. Agents know how to write high-quality software (they’ve seen “every possible permutation” in training); the job is to prune latent space — tell it which choices to make. A Jupyter prototype and a new DB index type warrant fundamentally different choices; owners must make legible what’s prototype vs production (staged rollout, A/B, feature flags), then give the agent tools to reason about which it’s doing — “give it bounds and context, but still give it the space to reason, be creative, and cook.”

9. All code is prompts. Every bit of text fed to the agent is prompting, which means all code in the repo — not just the docs — is prompts. Therefore unify the codebase on the same patterns: with one observability stack, the model translates context across the repo without loss; with six stacks it burns attention figuring out which is canonical. (Codebase consistency is context engineering.)

10. Three phases of context delivery over a PR.

• Grounding (start): the agent file holds a numbered set of steps for every rollout — ground in the docs knowledge-base + the ticket, spider ADRs/design docs to gauge blast radius, read the critical user journeys to keep a QA plan in mind. Accept slowness to page in global context.
• Messy middle: as the agent writes code and runs tests, exploit its tool calls to just-in-time prompt-inject and steer output back to baseline. Tests/lints for agents differ from those for humans (agents truncate tool-call outputs; respond well to descriptive error messages that point to runbooks). The canonical example: the missing-timeout-and-retry network outage — no eslint plugin exists, but you can “vibe a set of guardrails into place with 100% coverage + exhaustive table-driven tests, migrate the codebase in one go, and just-in-time surface this failure every time it writes a fetch call.” Tool-call output is weighted less during auto-compaction, so it doesn’t pollute the window up-front.
• Review and merge (end): the diff is now a static artifact, so use “very many LM-as-judge” against static guardrails; judges collaborate with the implementation agent over the PR thread (more text back) to realign the diff.

11. Review personas + coarse structural tests. Point the agent at a curated set of review personas (bulleted guardrails). Cheap team loop: a Slack thread about fixing a perf regression → @mention the agent → “put up a PR that adds it to our static guardrails.” Use blunt structural hammers: require every React component to have a snapshot test with 100% branch coverage → the model “naturally decomposes, makes pure, avoids prop-drilling, puts hooks close to the data” to satisfy it; statically ban any/unknown (except validated root/DB input) via eslint + require 100% typing/coverage → “weird type probing just falls out.” Failing checks that say why they failed and what to do let the agent self-heal.

12. Treat the agent as a teammate at review. It must convince you to merge — biased toward merge / benefit of the doubt, as with humans. You don’t shoulder-surf teammates; you take their word but ask for staging logs or a screenshot. Require the agent to do likewise: computer use / browser use / the Codex app (“fantastic, highly recommended”), or “vibe up Xvfb + a headless display in Docker + ffmpeg to record a reproduction video — Codex slings ffmpeg better than anybody in this room.” Reviewer agents are “really just a matrix CI job that points at a bunch of markdown files to judge”; drive remediations to completion.

13. Systematize feedback → “dream over it every night.” Every review comment, interruption, intervention, failed build, and production exception is a signal that context was missing. Slurp it all up and “dream over it every night, pointing a bunch of sub-agents at it” to distill better human prompting + missing guardrails — heading toward “more and more headless, less human-interrupt-dependent.” (He nods to the next talk — the dream-phase pattern.)

14. Vibe coding is part of success; the group-tech-lead operating mode. Many guardrails affect only local dev and the code “can be gross” — vibe coding makes it possible to stop caring about parts of the software-production function, letting you operate like a group tech lead / org lead: no visibility into every keystroke, caring instead about invariants, interfaces, and whether components “do what they say on the tin with high reliability.”

15. Q&A. Why shift right not left? Once requirements are surfaced at the right time, agents auto-discover the relevant persona-oriented guardrails by change category, so misbehavior patterns rarely appear; only when guardrails are needed across ~15 context windows (and get auto-compacted away) does he shift that one left. Practical implementation? He’s bringing these techniques to OSS — the Rust artichoke-rand-mt (Mersenne Twister) crate — using Codex-app automations to take his hands off maintenance (reviewer agents “coming”).

What was actually ingested

Full ASR transcript (~30 min, single un-chaptered block). The talk is slide-driven; the transcript captures the spoken narration only (slides not recovered). The fetch returned the transcript doubled; the duplicate was disregarded. Proper nouns cleaned at ingest (the ASR renders “Codex,” “Opus 4.5,” “GPT 5.2,” “ffmpeg,” “eslint,” “Artichoke/rand-mt” with some variability). The closing Q&A (two questions) is included.

Linked entities and concepts

• Concepts this source informs: agent-harness (the canonical first-person definition + the three-phase context-delivery model + shift-right + review-personas + LM-as-judge), agentic-engineering (the discipline: humans steer / agents execute; unblock the execution team), vibe-coding (vibing guardrails into place; gross local code; the group-tech-lead operating mode), automation-vs-augmentation (humans on steering/judgment, agents on execution; “more headless”), software-3.0 (all-code-is-prompts; prune latent space).
• Promoted to an entity this ingest (second-source rule — also the author of the OpenAI Codex harness-engineering blog): Ryan Lopopolo.
• Entity already in wiki: OpenAI (Lopopolo’s employer; Codex / Codex CLI / Codex app).
• Dangling (single-source mentions, deferred): AI Native Dev / AI Native DevCon (channel/venue), Artichoke / artichoke-rand-mt (his Rust OSS).

Source-to-source relationships

• Supports 2026-02-11-lopopolo-codex-harness-engineering — same author; talk-form articulation of the Codex blog case study (definition + three-phase model vs the blog’s operational invariants).
• Supports 2026-05-04-rethinking-agents-harness-is-all-you-need — the harness-is-the-durable-asset pole; the write-it-down → reviewer-agent → static-guardrail ladder is its practitioner mechanics.
• Supports 2026-05-08-bratanic-unified-agentic-memory-hooks — Lopopolo explicitly invokes the dream-phase (nightly sub-agent distillation of feedback signals).
• Supports 2026-05-06-bockeler-engineering-of-ai-agents-context-harnessing-autonomy — Böckeler relays the harness engineering name crediting the Codex team; this is that team’s own articulation.
• Note (no typed edge): Lopopolo sits firmly on the durable-harness pole, in tension with [[2026-06-11-kilpatrick-sequoia-model-eats-the-harness|Kilpatrick’s model eats the harness]] — but his “retool with every point release” is the same subtraction mechanism Kilpatrick names, and the wiki’s reconciliation (the durable asset is the discipline + guardrails + telemetry, not any specific harness component) covers it. Logged on the agent-harness Debates section rather than as a source edge.

Source-quality note

Auto-generated (ASR) transcript fetched via the youtube-transcript-skill (Playwright/DOM route). Channel AI Native Dev; the source is a recorded conference talk, so the spoken content is high-fidelity but slide visuals are absent. Transcript provenance (ASR) does not feed confidence per [§Lifecycle].