We sent the spec for our skill forge to four AI models for review, each working alone, none seeing the others' work. Three sent back critique. The fourth, Claude Opus 5.5 running in Cline, sent back its review and then asked whether it should also build the trigger-test harness it had just recommended. An hour later we had roughly 1,600 lines of dependency-free Python, 30 passing tests, and a delivery note that described its own limitations more precisely than most human contractors bother to. The metered cost of the whole session: $10.04. The four-reviewer comparison itself, and what we did with the six points all four agreed on, is its own post.
Disclosure: Wallaby Token operates an OpenAI-compatible inference API. We sell Kimi K3; Claude is not part of our public catalog, and we brought our own access for this work, running it through our own gateway so every token was metered like customer traffic. That metering is part of the point, as it was when we paid Claude to audit our blog for $2.50.
This is part 2 of our metered Claude series. Part 1 is the four-reviewer experiment this session came out of; part 3 is the $51.77 full-throttle refactor. Each one also stands alone.
The review, before the code
Worth noting first: the review itself was careful in a way we did not prompt. Before writing anything, the model searched the workspace for our actual implementation, and when it did not find the code, its review opened with a disclosure that everything below was based on the brief alone. No pretending to have read source it had not seen.
Its sharpest criticism of our design was that the most important maintenance hook lived inside the artifact it was supposed to monitor: if the artifact never loads, the hook never runs, and the failure ledger stays empty. Fair hit. Its top recommendation was an external trigger-test suite. Then it offered to build one.
What it built
One caveat before the anatomy: every green test mentioned below ran against a mock model, not a real one. What that means comes at the end, because it matters.
The harness it delivered is a small, opinionated piece of engineering:
- Prompt sets as data. Each test set is a JSON file: the skill under test, decoy skills the model might confuse it with, and pairs of prompts (one that should trigger the skill, one near-miss that should not), each with a one-line rationale.
- Seeded shuffling. The candidate list is reshuffled on every trial so a skill listed first cannot coast on position, but the shuffle is seeded, so two models or two description variants see identical orderings.
- Statistics that refuse flattery. Results come with 95% Wilson confidence intervals, because with 15–20 prompts per side, the difference between 15/20 and 17/20 is usually noise. Its words, not ours: small sets are noisy.
- Preflight cut-off. A missing API key or a malformed request stops the run before a single call is made, and nothing is written to disk. No burning budget to discover a typo.
- Honest accounting. Unparseable replies and failed calls are counted separately, never folded into the accuracy rates. Every raw response lands in a JSONL file with a fingerprint of the prompt set, so any report can be regenerated later.
How it worked
Three habits we did not ask for.
First, before writing a line, it read our existing tooling and matched its conventions: zero-dependency, standard library only, its stated reason being to follow our project's style. Second, it tested the preflight path by deliberately unsetting its own API key, verifying the run aborts before any money moves. Third, while cleaning up its tests, it chased down a ResourceWarning for an unclosed socket, the kind of warning most humans leave for a rainy day.
Thirty tests, all green. Total cost, $10.04.
Where the deliverable stops
The harness has only ever run against a mock model. There were no API keys in its environment, and its own delivery note says so plainly: the mock's numbers say nothing about quality, the suite tests one-turn routing only, and a small candidate list makes triggering easier than in a real library. We are quoting the model's own caveats because they are exactly right. A test harness validated against a stand-in is plumbing, not evidence. And like everything we publish from our gateway, this is a field observation under our own setup, not something an outside reader can reproduce as-is.
Since that night, we have ported the harness into our internal evaluation tooling, where all 30 of its tests still pass. Runs against real models, with real routing prompts, are a separate experiment with its own write-up coming; we will not borrow conclusions from numbers we do not have yet.
What $10.04 actually bought
Itemised: one design critique that found a structural flaw we had missed, and one working test harness implementing that critique's top recommendation, delivered with seeded shuffles, confidence intervals, and a manual.
We run an inference business, so we are biased here, but we think the most underpriced use of a frontier model right now is not generation. It is review with tools attached. Give a strong model your actual artifact and permission to build, and the deliverable starts looking less like feedback and more like a contractor's first invoice. A cheap one, at that.
The series, in order
- Four reviewers, one spec, six agreements: convergence is signal, not validation
- The $10 test harness (you are here)
- The $51.77 refactor: what full-throttle Opus actually buys
Run a session like this yourself: any OpenAI-compatible key, your tool of choice, an itemised receipt on the first call. $0.50 free credit; top-ups start at $20