What running it against real people did
v1 was reasoned into existence from source material. It looked coherent. Scoring it against three real workers broke it in five places, and a reader asking “what if someone scores low at the bottom and high at the top?” broke it in a sixth. Only one — the axis imbalance — was something I could have found by reading my own design.
The rest needed a person. A gate that conflated a practice with an artifact only shows up when someone does the practice without the artifact. A gate that asks you to rate a capability only shows up as a problem when the evidence for it is thin and the whole ladder swings on it.
Three cases is not a validation sample. Nothing here establishes that the levels are the right levels, that the thresholds are right, or that two people would score the same worker alike. No inter-rater check was run — every score is mine.
What three cases can do is expose gates that are unanswerable, wrongly placed, ambiguously worded, or duplicated. All five fixes below are of that kind: structural, not empirical. None was chosen to move a particular person's score.
The test I applied to each change: would this still be right if the case that revealed it didn't exist? If the answer was no, it didn't go in.
The reference cases
All three are in the wiki already, all three were observed doing the work rather than answering questions about it, and — importantly — two of them I had not been designing against. Blum's trajectory supplied nine of v1's gates, so he is the overfit risk. Nika and Carson are the controls.
Google AI product lead · Dec 2025
Five specialised tools stitched by hand across a 20-minute workflow — Perplexity, a custom GPT, v0, Veo, NotebookLM. Deliberate, skilled, and reusing her own PRD generator across projects.
No persistent context about her work, no connectors, nothing running without her. The human is the integration layer. The corpus treats her as the explicit contrast to Blum.
PM at Melio · Aug 2026
One harness holding 70–80% of screen time, an agent-maintained Notion board, three scheduled tasks, a context layer the agent repairs itself, friction logged inside skills, and an onboarding plugin the whole company uses.
Nine of v1's thirteen gates came from watching him. Any change that conveniently raises his score is suspect by construction.
Solo founder, Untangle · Aug 2026
10–15 concurrent cloud agents in a folder queue, playbooks as the reusable unit, a two-loop review ceiling and a narrated video walkthrough before any PR lands, and a deliberate three-tool division of labour.
No colleagues, so nothing to distribute to; no self-repairing context layer. Strong where Blum is strong, weak in different places — which is exactly what a discriminating instrument should show.
The defects
Each one names the case that surfaced it and the principle that justifies the change independently of that case.
v1's G9 asked whether you review output against criteria set in advance and whether those criteria are written down. Those are different things at different maturities. Blum reviews everything — it'll give me a draft and then I'll change it and I'll just send it
— but has nothing written. One gate, two verdicts, so the gate failed and pinned him at Level 3 despite passing every delegation marker emphatically.
The principle, independent of Blum: Level 4 is defined by retaining judgment while delegating tasks. Formalising that judgment so it can be handed over is a compounding practice — it belongs beside friction logging and distribution at Level 5, not beside scheduling at Level 4.
v1's G2 asked whether you notice when output is wrong in your own area — reliably, and I know where it tends to fail
. That is a self-rating of an internal capability, which is precisely what the model's own design note tells you not to do: ask for observable facts, not agreement statements. v1 violated its own rule in its second question.
The sensitivity check made it worse: because G2 gates Level 2, an error there drops any score to Level 1 regardless of everything above it. The most load-bearing item in the instrument rested on its least observable evidence.
The principle: at the bottom of a ladder, gates should test a low-bar behaviour that is either happening or not. Discrimination belongs higher up, where there is more to discriminate between.
v1's G7 asked whether AI work runs on a schedule. Carson's setup is emphatically delegated — Watchdog sweeps every customer account and returns the top three things going wrong; Land PR fires when an agent reports a PR done, runs up to two review loops, and records a narrated walkthrough. Neither is on a schedule. Land PR is event-triggered, and Watchdog he fires by hand when he feels the need.
The principle: the corpus's own loop taxonomy names four triggers — heartbeat, cron, hook, and goal. Only two of them are schedules. A gate testing “work starts without you” that recognises just one trigger type is measuring the implementation, not the property.
The whole point of the two-axis structure is that the axes compound — the source study's finding is that having both capabilities is worth more than the sum of having each. But v1 measured fluency with 4 items and work-system design with 9. Fluency scores moved in 12.5-point jumps, and the imbalance flag needed a 34-point gap it could rarely reach.
v1 acknowledged this in a note — “most observable markers live on the S axis, which is why the assessment asks more about it” — which was honest and also a description of a defect rather than a justification for one. The fix is not padding with unobservable questions; it is finding observable fluency markers, which exist.
Once G9 was split, its written-criteria half and v1's G12 — “you could hand someone else a written definition of what a good result looks like” — were the same gate under two names, both at Level 5, both on the fluency axis. Scoring Blum against both produced the same verdict from the same evidence twice, which inflates whichever axis carries the duplicate.
The principle: duplicated gates are silent weighting. If a construct matters double, weight it deliberately rather than by accident of drafting.
Strict gating is the right rule — it is what makes this a diagnostic that names one next action rather than a score. But “Level 3” alone threw away the most informative thing about the Blum profile, which was that every Level 5 gate was already met. That lived in a flag below the fold.
v2 reports a level and a reach: the highest level whose own gates are all met, ignoring gaps beneath it. When they differ, the headline says so.
Nothing stops someone failing both Level 2 gates while meeting every gate at Level 4 — an employee who inherited a configured system, or anyone who read a low gate more strictly than a high one. Strict gating puts them at Level 1, which is right. But everything wrapped around that number was wrong: the page told a person with scheduled tasks writing to a system of record that nothing accumulates — what you worked out today will not be there tomorrow
, and advised them to start saving their prompts.
The principle: a level is a summary, and a summary that contradicts the answers it summarises is worse than none. Where the profile and the level's narrative disagree, the narrative yields — not the gating.
The fix separates two causes the instrument had been conflating. Some gates are true by construction if a higher one is: a defined checkpoint on delegated output (G9) means you check output before using it (G2); written review criteria (G13) mean the review happens (G9). When the higher holds and the lower doesn't, that is far likelier a misread question than a real gap, and it is now reported as one — with both answers quoted back, because the level turns on it. When no implication applies and the inversion is real, it gets its own diagnosis: automation you did not build — delegation inherited without the habits underneath it. That is not a contradiction. It is the most exposed profile on the scale, and it had no name before.
Before and after
v1 scores are as originally computed. v2 scores were produced by driving the live instrument, not by hand — the numbers below came out of the page.
| Case | v1 | v2 | F / S (v2) | Open gates at the next level |
|---|---|---|---|---|
| Marily Nika | L2 | L2 | 67% / 17% | G3, G4, G5 — persistent context, share of work, tool reach. Unchanged by the calibration, which is the point: the control case at the low end did not drift. The 50-point axis gap now trips the imbalance flag correctly — high fluency, no system to accumulate it in. |
| Daniel Blum | L3 | L4 | 92% / 100% | G13 — written, handable criteria. Moves up one level because the gate that blocked him was conflating two things and sitting too high. He is still blocked, by the same underlying weakness, at the level where it belongs. The v1 “thin base” flag no longer fires — it was an artifact of the misplaced gate. |
| Ryan Carson | — | L4 | 100% / 78% | G11, G12, G14 — self-repairing context, automatic friction capture, distribution. Not scored under v1. Lands at the same level as Blum with a completely different set of open gates: Blum's weakness is formalised judgment, Carson's is that nothing improves itself and there is nobody to hand it to. Two people, same rung, different advice. |
The Carson row is the strongest evidence the instrument is doing real work. Under a scoring scheme that only counted totals, he and Blum would be interchangeable. Under gates, they are the same level for opposite reasons, and the next action each should take is different.
Still uncalibrated
Listing these is part of the calibration. An instrument that doesn't say where it is unvalidated invites being used as though it isn't.
Five rungs is a borrowed shape, taken from a firm-level model with a 2022 survey base. Nothing tested whether workers actually cluster into five groups rather than three or seven.
Every score here is mine, from transcripts, using my own reading of my own gates. The standard check — two independent raters, agreement measured — has not been run. This is the largest gap.
All three cases were scored from observation. The instrument is designed to be self-administered. Those are different measurements and nothing here connects them.
“More than 40%” of screen time, a 34-point axis gap, three options per gate — all chosen by judgment. Blum clears the 40% band by double; nothing establishes that 40 is the right line.
The firm-level source locates its value inflection between stages 2 and 3. No equivalent has been established for workers. Levels currently describe; they do not yet forecast.
All three cases are a PM, a PM, and a solo founder — screen-based, autonomous, tool-rich. The gates assume you can choose your own tools and restructure your own work. Many people cannot.