Last week we sent the same product spec, for an internal tool we are developing, to four AI models for review, on the same night, each working in isolation. None saw the brief we gave the others; none saw another's review. When we laid the four write-ups side by side the next morning, six criticisms appeared in all four. And on one architectural question, two reviewers flatly contradicted each other, in terms worth quoting.
This is how we read a multi-model review: convergence as signal, disagreement as a decision you still have to make yourself, and one boundary we think the whole conversation about "AI review" keeps blurring. Four AI reviews are four opinions, not four validations.
This is part 1 of our metered Claude series: three posts from one week of running Claude through our own gateway with the meter on. Part 2 is the $10 review that built its own test harness; part 3 is the $51.77 full-throttle refactor. Each one also stands alone.
The setup
One English brief, frozen before dispatch. Four reviewers, same night, no cross-contamination. Three ran on free public endpoints of varying capability tiers, deliberately unnamed here; the fourth was Claude Opus 5.5 with tool access, metered through our own gateway: $10.04, and it ended up building the test harness it recommended, which is its own story. The point of the design was independence. If four models with different training lineages hit the same weak spot, that spot is probably real. And the mixed tiers are a feature, not a flaw we hid: agreement across unequal models points at defects in the spec itself rather than one model's bias.
Where all four converged
Six points, independently raised by every reviewer:
- The problem is real but niche. One reviewer's phrasing: the best first customer is not someone who wants to build the thing our spec describes, it is someone who has already been burned by the same failure twice. That single sentence is rewriting our positioning.
- Templates are not a moat. "Copyable in an afternoon," one wrote; another noted the core checks could be redone "in about an hour, mostly with regex." Both are right, and both pointed at the same consequence: the durable asset has to be accumulated evidence, not the prompts.
- "Quality is model-independent" is an overclaim. One put it crisply: your prompt sets an upper limit on quality, not a lower one. Weaker models are exactly the ones most likely to ignore a long spec.
- The feedback ledger will idle, an existential risk. Agent self-reporting is unreliable; one reviewer traced it precisely: the feedback hook lives inside the artifact, so if the artifact never loads, the hook never runs, and the ledger stays empty forever.
- Health checks are heuristics, and must be labelled as such. "Never let 'healthy' imply 'works correctly.'" A green report on structure proves nothing about behavior.
- Don't build the web wizard. All four deprioritised the form-based UI we had sketched. Nobody leaves their agent to fill in a form.
Four independent reviews, zero coordination, six identical hits. We treated every one as actionable, and most are already design changes.
Where they fought
On one question the reviews split cleanly, and we print the disagreement because it is more instructive than the consensus. The topic: where the system's depth should live.
Reviewer B argued the protocol integration is the whole game: in its words, the only form that closes the autonomous feedback loop. Reviewer D argued the opposite. That same integration, it wrote, is "the wrong home" for the core content, which should ship as portable text, with code reserved for what code must do.
Both are defensible. Neither can be settled by a fifth opinion. We ruled based on our own constraint: distribution friction beats architectural elegance for a tool nobody has installed yet. The answer we shipped is a split, portable content first, protocol integration as the resident form for existing users. The disagreement did not resolve the question for us. It mapped the trade-off, and mapping was the actual deliverable.
The blind spots that stung
Beyond the scored findings, three observations caught things our design simply had not seen:
- Mandatory sections force confabulation. Require a "known pitfalls" chapter in every new artifact, and a model with no failure history will invent plausible-looking pitfalls. Invented authority is worse than an empty section. We have watched this happen in our own trigger experiments: the unassisted baseline model fabricated a pitfall, and a price that does not exist.
- The one-use abandonment profile. A user reads the health report, agrees with every line, then just edits the file by hand, and the tool has failed silently. We had no metric for "time to verified improvement." Now we do.
- One reviewer went looking for the code before answering. It searched its workspace for our implementation, found none, and opened its review by disclosing that everything below was based on the brief alone. Of the four, it was the only one that tried to check the actual car before reviewing the brochure. There is a lesson in that about what to feed reviewers, and about which critiques to weight.
How we triaged
Every finding went into one of two buckets. Ten items were adopted outright; they cost little and were self-evidently correct once stated (confidence intervals in every reported metric; abort before any paid call when configuration is broken; count garbage separately from errors). Eight more were parked behind internal experiments with pre-agreed criteria, because they were plausible but untested, including the one existential question a reviewer raised about whether the artifact is even the right unit of governance.
That split is the part we would recommend to anyone running this play: adopt the cheap truths immediately, and refuse to adopt the expensive ones on the strength of rhetoric, however fluent.
The boundary: convergent opinion is not validation
Here is the discipline we now apply to ourselves, and the one we think matters most. Evidence comes in kinds: a design review reads your materials and argues; a code review inspects the implementation; a behavioral experiment runs the thing under fixed conditions and measures; real-user validation watches someone with an actual problem use it. Each kind answers a different question, and none can substitute for another.
Four AI reviews, however independent, however sharp, all sit in the first row. They converged on six real weaknesses, and we acted on them. But "four models agree" is opinion converging, not evidence accumulating. The next round for this project is behavioral tests with pre-registered fail criteria and a pilot with real users, and we intend to publish both, wins or losses. We have form for publishing the losses.
If you run multi-model reviews of your own work, and for the price of a metered session you should, treat the output as a map of where to dig, not as ground truth. The models can find your weak spots with unsettling reliability. Deciding what to do about them remains, stubbornly, your job.
The series, in order
- Four reviewers, one spec, six agreements (you are here)
- The $10 test harness: the reviewer that built what it recommended
- The $51.77 refactor: what full-throttle Opus actually buys
Beyond the series: the spec under review was for our skill forge, whose own experiment report is now public.