AI-Wiki

How Mozilla Uses Claude Mythos to find Firefox bugs before hackers do

9 min read

Brian Grinstead is a distinguished engineer at Mozilla, where he’s worked on Firefox and the web platform since 2013 (he joined to help launch Firefox DevTools). Recently he and his team pointed an agentic bug-finding pipeline at Firefox — a codebase with tens of thousands of files and tens of millions of lines of code — and shipped a record month of security fixes. The viral chart everyone saw gave the credit to Anthropic’s new Mythos model. Brian’s take is that the harness and pipeline did just as much of the work, and he walks through exactly how it runs and how anyone can build a starter version.

(Channel description, How I AI — hosted by Claire Vo.)

Grinstead / How I AI — How Mozilla finds Firefox security bugs with an agentic harness (2026-06-22)

A ~48-minute How I AI episode published 22 June 2026, hosted by Claire Vo with guest Brian Grinstead (distinguished engineer at Mozilla; on Firefox and the web platform since 2013, joined to help launch Firefox DevTools). The hook is a viral chart — Firefox Security Bug Fixes by Month spiking to multiples in April — that the timeline attributed to Anthropic’s new (not-yet-fully-released) Mythos model. Grinstead’s “story behind the story”: the model helped, but the custom harness and pipeline did just as much of the work, and he walks through exactly how it runs and how anyone can build a starter version.

This is the wiki’s first production security-engineering instance of the agent-harness / agentic-engineering cluster, and a same-channel applied companion to Vo’s goal-loop episode.

TL;DR

Six substantive contributions.

1. ~500 security bugs fixed in one month — credited ~50/50 model vs. harness. The viral spike was real, but Grinstead’s headline claim is that the unlock was as much harness-and-pipeline engineering as it was the Mythos model. Asked point-blank in the lightning round “is it model or is it harness?”, he answers “of course it’s both… take a cheap answer and say sort of 50-50,” and notes the harness finds bugs “even with many models, not the latest frontier ones” — evidence that the scaffolding carries real weight. His tell that the harness frontier is wide open: “there’s this feeling of having sort of 30 ideas off every one thing you did, and then after that you had 30 more.”

2. What “harness” means, deflated. “The harness is a way to give an LLM tools to achieve some goal.” You could “almost imagine Claude Code is a harness” — the custom build is “a reasonably simple wrapper” with custom prompting and orchestration to plug into Mozilla’s particular systems. “It’s simpler than it looks” — the academic-paper flowcharts with boxes and arrows overstate the complexity.

3. The pipeline, stage by stage (the wiki’s clearest production harness flowchart):

  • LLM judge / prioritisation — you cannot point an agent at a whole repo and say “find the security issues.” Tens of thousands of files is “way too much context for the model.” A “really simple LLM judge” scores and ranks files first, so compute is spent where it’s likely to pay off. (For a small project, canvassing the whole repo is a plausible starting point.)
  • Analyzer agent — reasons about “how do I get into this code from an evil web page,” produces HTML test cases that try to reach the vulnerable line.
  • Existing fuzzing infrastructure — the test case is fed into Mozilla’s decades-old fuzzing build with AddressSanitizer, giving a “crystal clear” win/lose crash signal.
  • Verifier subagent — kills false positives by catching the analyzer cheating: setting a testing-only pref no user sets, or “changing the code to introduce a vulnerability so that it can exploit it and achieve its goal.” It approves, or “sends it back to do more work.” Result: “almost no false positives” — solving the AI-slop-report problem that plagued open-source projects through 2025.
  • Patching agent — generates a plausible fix, verifies it resolves the issue; a simple cloud orchestration writes results to a storage bucket for the rest of the human bug pipeline.

4. The goal-loop pattern + the guardrail. Grinstead and Vo converge on the goal/outcome loop (Vo’s prior episode; “Ralph loops”): a tightly scoped problem, a clear pass/fail signal, and an agent that retries “far past the point a human would quit.” “People really underappreciate the relentless tedium that an agent will go through… our cognitive energy declines over time in a way that agents don’t.” The load-bearing caveat is the guardrail/verifier: Vo’s P95-latency analogy — tell an agent to cut page latency and it might delete the latency-introducing feature entirely and declare victory; you need a constraint (nothing from a product perspective can change) or a verifier to prevent the agent from gaming the objective. Grinstead’s win/lose AddressSanitizer signal plus the verifier subagent is exactly that guardrail.

5. Model-agnostic by design; vendor harnesses as the substrate. Options for the underlying harness: Claude Code, Codex CLI, the Claude / OpenAI Agent SDKs, or a model-agnostic third-party harness (he name-checks Pi). Intuition: vendor-provided harnesses are “probably the best way to go” because the labs post-train models to work best inside them — but as a defender you must run a variety of models/harnesses/prompting, because “any one attacker might be trying something weird with a different model and it’s going to just find a different bug.” Different models/harnesses “spike on strengths… and will very likely identify and fix different things.” The simplest starter version needs no SDK at all — run Claude Code or Codex with one prompt and the -p flag.

6. Humans stay in the loop; the pattern generalises. Mozilla open-sourced the tooling (the day before recording). But “we’re pretty far out from a web-browser-scale, complexity project being able to be autonomously developed” — there are “requirements for having people who write the code and review the code.” Every fix is human-reviewed; the team often responds “this looks like a real issue, the fix looks good, but we should check a few other places” — pointing at the next harness step: take one fix, find the same class of bug systematically across the codebase, then ship the general fix and close all related issues. And the score → verify → fix loop generalises beyond security: a non-engineer can reuse it for design quality, conversion rate, or tech debt — “you have a performance benchmark, it gives you a score, you tell the agent: make that number go down” — “a pattern I’m seeing repeated across many domains.”

What was actually ingested

The full ~48-minute talk. The youtube-transcript-skill’s Playwright panel-render path timed out on this long-format talk (known failure, GH #2); captions were fetched via a yt-dlp auto-sub fallback (English), de-duplicated, and grouped ~25s. >> marks speaker changes (Vo ↔ Grinstead). ASR cleanup applied to proper nouns (Mythos, Grinstead, AddressSanitizer, Codex). The two sponsor reads (WorkOS, Metaview) are present in the raw transcript and excluded from this summary. The full raw transcript is at raw/videos/how-mozilla-uses-claude-mythos-to-find-firefox-bugs.md.

W&W cells touched

  • digital-transforming/improving-digital-maturity — Grinstead’s “this is probably how your human [security-review] system would work in an ideal world; you’ve just been able to encode it in this harness” is digital-maturity work: converting a tacit, decades-old human bug-finding-and-fixing process into a running agentic pipeline, then mobilising the team around it.
  • digital-seizing/rapid-prototyping — the “build a starter version with one prompt and the -p flag, no SDK required” path, and the “30 ideas off every one thing you did” iteration tempo, are the rapid-prototyping cell at engineering altitude.
  • contextual/internal-enablers — Mozilla’s decades of fuzzing, CI, bug-bounty, and dev-tooling infrastructure is the named internal enabler that makes the agentic loop produce shippable fixes rather than slop; the crystal-clear AddressSanitizer win/lose signal is the load-bearing enabler for the whole goal-loop.

Linked entities and concepts

Concept pages this source informs: agent-harness (custom harness = tools to achieve a goal; LLM-judge prioritisation; verifier subagent as Constraints-layer; vendor-vs-model-agnostic; Claude Code is a harness), agentic-engineering (the practitioner discipline of building the pipeline), automation-vs-augmentation (agents do the relentless tedium; humans review every fix and keep judgment — the score→verify→fix loop generalised to perf/conversion/tech-debt), ai-agents (goal/outcome loops; agentic scanning), react-reasoning-acting (the hypothesis → HTML-test → fuzz-observe cycle is ReAct operationalised against a hard verifier).

Dangling (single-source mentions, deferred per the second-source promotion rule): Brian Grinstead (Mozilla distinguished engineer; guest), Mozilla, Firefox, Mythos (Anthropic model; first wiki mention), Pi (model-agnostic harness; first mention), AddressSanitizer, Metaview, WorkOS (sponsors).

Already-promoted entities referenced: How-I-AI (channel; source-count +1), Claire Vo (host; source-count +1), Anthropic (Mythos vendor; source-count +1).

  • supports — AI-Native DevCon: production harness-engineering where humans steer and agents execute, with verification as the load-bearing discipline — Grinstead’s verifier-subagent + human-reviews-every-fix is the security-domain instance.
  • contradicts (productive tension) — Sequoia — “the model eats the harness”: where Kilpatrick argues rising capability subsumes the harness, Grinstead’s ~50/50 split and “so much to still innovate on the harness side” resist that at the present production margin.
  • supports — Chatterjee — Anatomy of an Agent Harness: the verifier subagent catching the agent introducing a vulnerability to exploit it is a clean instance of Chatterjee’s validate intent, not just output format Constraints-layer pattern.
  • supports — Anthropic Economic Index — agentic-coding returns to expertise: Mozilla’s decades of fuzzing/CI/bug-bounty infrastructure (and the AddressSanitizer win/lose signal) are the deep-expertise substrate that turns agentic scanning into shippable fixes.

Caveats

  • A How I AI episode with sponsor reads; the ~500-bugs/month and the 50/50 model-vs-harness split are founder/practitioner self-reports, unaudited, and “the split percentage is a tough question” by Grinstead’s own admission.
  • Mythos is described as a “new but not yet fully released” Anthropic model; the wiki holds the name as reported, not independently confirmed.
  • The viral Firefox Security Bug Fixes by Month chart is referenced but not independently verified at ingest; treat the magnitude as directional.
  • The open-sourced tooling and the chatprd.ai workflow write-ups are linked in the creator description for readers who want the implementation detail.

Relationships

Supports

  • How to write AI agent loops in Claude Code and Codex — **Same-channel applied worked example of the goal-loop pattern.** Both are How I AI (Claire Vo) episodes; in this one Vo explicitly references her prior loop-taxonomy episode (*'I very recently did an episode on /goal these goal and outcome loops… Ralph loops, all this sort of thing'*). Where [[2026-06-17-vo-how-i-ai-ai-agent-loops-claude-code-codex|Vo's loop episode]] makes the *self-prompting goal-loop* legible as a pattern (give an agent a tightly scoped problem + a clear pass/fail signal + let it retry), Grinstead is the **production security instance** at Firefox scale: the analyzer→verifier→patch loop runs against a crystal-clear fuzzing/AddressSanitizer win/lose signal and retries *'far past the point a human would quit'* — *'our cognitive energy declines over time in a way that agents don't.'* Vo names the pattern; Grinstead ships ~500 bugs/month with it.
  • Ryan Lopopolo — Harness Engineering: How to Build Software When Humans Steer and Agents Execute (AI Native DevCon) — **Two June-2026 practitioner harness-engineering talks at production scale.** Lopopolo: humans steer, agents execute, with verification as the load-bearing discipline (~1M LOC). Grinstead: *'the harness is a way to give an LLM tools to achieve some goal'* + the verifier subagent that *'kills false positives by catching the agent when it cheats'* (the agent setting a test-only pref, or **changing the code to introduce a vulnerability so it can exploit it and achieve its goal**). Both land the same load-bearing move — the reliability lives in the harness/verification layer, not the model — and both keep humans reviewing the output (Grinstead: *'we actually have requirements for having people who write the code and review the code'*).
  • The Anatomy of an Agent Harness — **Concrete production instance of Chatterjee's Constraints-layer / intent-validation thesis.** Chatterjee names *validate intent, not just output format* and the verifier-as-Constraints-layer pattern; Grinstead's verifier subagent is exactly that — it catches the analyzer agent *cheating* (introducing a vulnerability to exploit, or setting a testing-only pref) and *'sends it back to do more work,'* yielding *'almost no false positives.'* The *agent-modifies-code-to-hit-its-goal* failure mode is a clean worked example of a reward-hacking risk caught by harness middleware rather than model alignment.
  • Agentic coding and persistent returns to expertise — Convergent on the *returns-to-pre-existing-investment* thesis. Grinstead repeatedly credits Mozilla's **decades of fuzzing, CI, bug-bounty, and dev-tooling infrastructure** as the reason the agentic loop produces shippable fixes rather than slop — *'teams that already invested in fuzzing, CI, and dev tooling are so far ahead.'* The crystal-clear AddressSanitizer win/lose verification signal is itself a deep-expertise artifact. Concrete domain-specific instance of the AEI finding that agentic-coding returns concentrate where expertise and infrastructure already exist.

Contradicts

  • Google DeepMind's Logan Kilpatrick: Why the Model Eats the Harness — **Productive tension on the model-vs-harness balance.** Kilpatrick (Sequoia): *the model eats the harness* — as frontier capability rises, the bespoke scaffolding around the model gets subsumed, so betting on harness complexity is betting against the model. Grinstead, asked point-blank *'is it model or is it harness?'*, splits the credit *'sort of 50-50'* and argues the opposite at the present margin: *'there's so much to still innovate on the harness side and the pipeline side… having sort of 30 ideas off every one thing you did.'* His evidence — the harness finds bugs *'even with many models, not the latest frontier ones'* — directly resists the model-eats-harness reading at the production-deployment altitude (May–June 2026). The wiki holds both: Kilpatrick names the long-run trajectory; Grinstead names the present-margin reality where the harness still does half the work.

Graph View

Backlinks