Calibration record — v1 → v2.
1 Sep 2026 · 3 scored cases + 1 adversarial profile · 7 defects fixed · structural calibration only

What running it against real people did

Seven defects, and six were invisible from the design alone.

v1 was reasoned into existence from source material. It looked coherent. Scoring it against three real workers broke it in five places, and a reader asking “what if someone scores low at the bottom and high at the top?” broke it in a sixth. Only one — the axis imbalance — was something I could have found by reading my own design.

The rest needed a person. A gate that conflated a practice with an artifact only shows up when someone does the practice without the artifact. A gate that asks you to rate a capability only shows up as a problem when the evidence for it is thin and the whole ladder swings on it.

The Blum scoring's headline finding — “built high on a thin base” — turned out to be an artifact of a misplaced gate, not a fact about him. Fixing the gate made the flag stop firing. That is what calibration is for.

What this is not

Three cases is not a validation sample. Nothing here establishes that the levels are the right levels, that the thresholds are right, or that two people would score the same worker alike. No inter-rater check was run — every score is mine.

What three cases can do is expose gates that are unanswerable, wrongly placed, ambiguously worded, or duplicated. All five fixes below are of that kind: structural, not empirical. None was chosen to move a particular person's score.

The test I applied to each change: would this still be right if the case that revealed it didn't exist? If the answer was no, it didn't go in.

The reference cases

Three profiles, chosen to span the range

All three are in the wiki already, all three were observed doing the work rather than answering questions about it, and — importantly — two of them I had not been designing against. Blum's trajectory supplied nine of v1's gates, so he is the overfit risk. Nika and Carson are the controls.

Control · low end

Marily Nika

Google AI product lead · Dec 2025

Five specialised tools stitched by hand across a 20-minute workflow — Perplexity, a custom GPT, v0, Veo, NotebookLM. Deliberate, skilled, and reusing her own PRD generator across projects.

No persistent context about her work, no connectors, nothing running without her. The human is the integration layer. The corpus treats her as the explicit contrast to Blum.

Derivation case · overfit risk

Daniel Blum

PM at Melio · Aug 2026

One harness holding 70–80% of screen time, an agent-maintained Notion board, three scheduled tasks, a context layer the agent repairs itself, friction logged inside skills, and an onboarding plugin the whole company uses.

Nine of v1's thirteen gates came from watching him. Any change that conveniently raises his score is suspect by construction.

Control · high end, different shape

Ryan Carson

Solo founder, Untangle · Aug 2026

10–15 concurrent cloud agents in a folder queue, playbooks as the reusable unit, a two-loop review ceiling and a narrated video walkthrough before any PR lands, and a deliberate three-tool division of labour.

No colleagues, so nothing to distribute to; no self-repairing context layer. Strong where Blum is strong, weak in different places — which is exactly what a discriminating instrument should show.

The defects

What broke, and what each fix rests on

Each one names the case that surfaced it and the principle that justifies the change independently of that case.

D1
Found by · Blum

One gate asked two questions and sat a level too high

v1's G9 asked whether you review output against criteria set in advance and whether those criteria are written down. Those are different things at different maturities. Blum reviews everything — it'll give me a draft and then I'll change it and I'll just send it — but has nothing written. One gate, two verdicts, so the gate failed and pinned him at Level 3 despite passing every delegation marker emphatically.

The principle, independent of Blum: Level 4 is defined by retaining judgment while delegating tasks. Formalising that judgment so it can be handed over is a compounding practice — it belongs beside friction logging and distribution at Level 5, not beside scheduling at Level 4.

Was — G9 (L4): “You review AI output against criteria you set in advance… and the criteria are written down.” Now — split. G9 (L4, F): delegated output passes a review step before it gets used. G13 (L5, F): the criteria you review against are written down, specific enough to hand to someone else.
D2
Found by · Blum, via the sensitivity check

A Level 2 gate asked you to rate a capability, and could collapse the ladder alone

v1's G2 asked whether you notice when output is wrong in your own area — reliably, and I know where it tends to fail. That is a self-rating of an internal capability, which is precisely what the model's own design note tells you not to do: ask for observable facts, not agreement statements. v1 violated its own rule in its second question.

The sensitivity check made it worse: because G2 gates Level 2, an error there drops any score to Level 1 regardless of everything above it. The most load-bearing item in the instrument rested on its least observable evidence.

The principle: at the bottom of a ladder, gates should test a low-bar behaviour that is either happening or not. Discrimination belongs higher up, where there is more to discriminate between.

Was — G2: “You notice when output is wrong in your own area, and can say what is wrong with it.” Now — “You check AI output against something other than your own impression before using it for anything that matters.” A behaviour, not a self-assessment, and one that no longer swings the whole result on an inference.
D3
Found by · Carson

A gate's wording excluded half of what it was meant to catch

v1's G7 asked whether AI work runs on a schedule. Carson's setup is emphatically delegated — Watchdog sweeps every customer account and returns the top three things going wrong; Land PR fires when an agent reports a PR done, runs up to two review loops, and records a narrated walkthrough. Neither is on a schedule. Land PR is event-triggered, and Watchdog he fires by hand when he feels the need.

The principle: the corpus's own loop taxonomy names four triggers — heartbeat, cron, hook, and goal. Only two of them are schedules. A gate testing “work starts without you” that recognises just one trigger type is measuring the implementation, not the property.

Was — G7: “Some AI work runs on a schedule, without you starting it.” Now — “Some AI work starts without you — on a schedule, or fired by an event such as a finished task or an incoming message.”
D4
Found by · reading the design

Two co-equal axes measured with four questions against nine

The whole point of the two-axis structure is that the axes compound — the source study's finding is that having both capabilities is worth more than the sum of having each. But v1 measured fluency with 4 items and work-system design with 9. Fluency scores moved in 12.5-point jumps, and the imbalance flag needed a 34-point gap it could rarely reach.

v1 acknowledged this in a note — “most observable markers live on the S axis, which is why the assessment asks more about it” — which was honest and also a description of a defect rather than a justification for one. The fix is not padding with unobservable questions; it is finding observable fluency markers, which exist.

Was — 4 fluency items, 9 system items. Now — 6 and 9. Two new fluency gates, both artifact-based rather than self-rated: G10 (L4) “you keep some record of where AI has failed you,” and G15 (L5) “you can name what your setup is specifically bad at.”
D5
Found by · the D1 fix

Two gates were asking the same question

Once G9 was split, its written-criteria half and v1's G12 — “you could hand someone else a written definition of what a good result looks like” — were the same gate under two names, both at Level 5, both on the fluency axis. Scoring Blum against both produced the same verdict from the same evidence twice, which inflates whichever axis carries the duplicate.

The principle: duplicated gates are silent weighting. If a construct matters double, weight it deliberately rather than by accident of drafting.

Was — G9's “written down” half and G12, separately. Now — merged into a single G13. Level 5, fluency axis, one verdict.
D6
Reporting change · not a gate

The headline number hid the shape

Strict gating is the right rule — it is what makes this a diagnostic that names one next action rather than a score. But “Level 3” alone threw away the most informative thing about the Blum profile, which was that every Level 5 gate was already met. That lived in a flag below the fold.

v2 reports a level and a reach: the highest level whose own gates are all met, ignoring gaps beneath it. When they differ, the headline says so.

Now — “You are at Level 3 · gates met through Level 5”. Strict gating unchanged; the shape is no longer buried.
D7
Found by · a reader, not a case

An inverted ladder produced a confidently wrong answer

Nothing stops someone failing both Level 2 gates while meeting every gate at Level 4 — an employee who inherited a configured system, or anyone who read a low gate more strictly than a high one. Strict gating puts them at Level 1, which is right. But everything wrapped around that number was wrong: the page told a person with scheduled tasks writing to a system of record that nothing accumulates — what you worked out today will not be there tomorrow, and advised them to start saving their prompts.

The principle: a level is a summary, and a summary that contradicts the answers it summarises is worse than none. Where the profile and the level's narrative disagree, the narrative yields — not the gating.

The fix separates two causes the instrument had been conflating. Some gates are true by construction if a higher one is: a defined checkpoint on delegated output (G9) means you check output before using it (G2); written review criteria (G13) mean the review happens (G9). When the higher holds and the lower doesn't, that is far likelier a misread question than a real gap, and it is now reported as one — with both answers quoted back, because the level turns on it. When no implication applies and the inversion is real, it gets its own diagnosis: automation you did not build — delegation inherited without the habits underneath it. That is not a contradiction. It is the most exposed profile on the scale, and it had no name before.

Was — Level 1, “nothing accumulates”, and advice to save your prompts — regardless of what was met above. Now — the level is unchanged, the narrative is replaced, and the next actions become the specific open gates beneath your reach. Two new flags: a probable-misread warning keyed to five implication pairs, and the inherited-automation diagnosis.

Before and after

The three cases, rescored

v1 scores are as originally computed. v2 scores were produced by driving the live instrument, not by hand — the numbers below came out of the page.

Casev1v2F / S (v2)Open gates at the next level
Marily Nika L2 L2 67% / 17% G3, G4, G5 — persistent context, share of work, tool reach. Unchanged by the calibration, which is the point: the control case at the low end did not drift. The 50-point axis gap now trips the imbalance flag correctly — high fluency, no system to accumulate it in.
Daniel Blum L3 L4 92% / 100% G13 — written, handable criteria. Moves up one level because the gate that blocked him was conflating two things and sitting too high. He is still blocked, by the same underlying weakness, at the level where it belongs. The v1 “thin base” flag no longer fires — it was an artifact of the misplaced gate.
Ryan Carson — L4 100% / 78% G11, G12, G14 — self-repairing context, automatic friction capture, distribution. Not scored under v1. Lands at the same level as Blum with a completely different set of open gates: Blum's weakness is formalised judgment, Carson's is that nothing improves itself and there is nobody to hand it to. Two people, same rung, different advice.

The Carson row is the strongest evidence the instrument is doing real work. Under a scoring scheme that only counted totals, he and Blum would be interchangeable. Under gates, they are the same level for opposite reasons, and the next action each should take is different.

Still uncalibrated

What three cases cannot tell you

Listing these is part of the calibration. An instrument that doesn't say where it is unvalidated invites being used as though it isn't.

Whether the levels are the right levels

Five rungs is a borrowed shape, taken from a firm-level model with a 2022 survey base. Nothing tested whether workers actually cluster into five groups rather than three or seven.

Whether two raters agree

Every score here is mine, from transcripts, using my own reading of my own gates. The standard check — two independent raters, agreement measured — has not been run. This is the largest gap.

Whether self-scoring matches observed scoring

All three cases were scored from observation. The instrument is designed to be self-administered. Those are different measurements and nothing here connects them.

Where the thresholds should sit

“More than 40%” of screen time, a 34-point axis gap, three options per gate — all chosen by judgment. Blum clears the 40% band by double; nothing establishes that 40 is the right line.

Whether the levels predict anything

The firm-level source locates its value inflection between stages 2 and 3. No equivalent has been established for workers. Levels currently describe; they do not yet forecast.

Whether it survives non-knowledge work

All three cases are a PM, a PM, and a solo founder — screen-based, autonomous, tool-rich. The gates assume you can choose your own tools and restructure your own work. Many people cannot.