One evening last week we pointed Claude Opus 5.5, running at full reasoning capacity, at a 1,203-line operations script we rely on daily, and asked for a refactor. About an hour later it delivered 1,570 rewritten lines, evidence that the old and new versions behave identically across 8 seeded scenarios, 17 passing unit tests, and a changelog that listed its own five out-of-scope decisions for a human to rule on. The metered bill: $51.77. The same class of task on Kimi K3 runs roughly $15–25 on our meter, but that figure is an extrapolation from our four-tier effort experiment, not a head-to-head rerun of this refactor. This post is the account of what the difference bought. Two days earlier, the same model had turned a $10 spec review into a working test harness.
Disclosure: Wallaby Token operates an OpenAI-compatible inference API. We sell Kimi K3; Claude is not part of our public catalog, and we brought our own access, metered through our own gateway exactly like customer traffic.
This is part 3 of our metered Claude series. Part 1 is the four-reviewer experiment; part 2 is the $10 test harness. Each one also stands alone.
The setup, and why we dared
The script is internal tooling: it rolls up project state, and a bad rewrite would corrupt records we actually use. So the session's ground rule came before any code. The model was told to build an equivalence harness first and treat behavioral identity as machine-checkable, not reviewable. No "looks right to me." Diffs or it didn't happen.
The cost curve
The meter readings tell the story better than a summary can:
- $14.31: orientation done; it starts on the largest function, a 300-line daily rollup
- $21.31: first context compression kicks in (112.5k down to 109k); the big function lands in four pieces
- $28.03: tail sections being rewritten block by block; the meter starts accelerating
- $42.02: it builds its own verification rig, an equivalence differ with 8 seeded runs plus a branch-coverage checker
- $48.27: the differ's internals: mocked external calls, per-file and per-log comparisons
- $51.77: final stretch: a static checker for undefined names, then a hard-boundary grep pass against every constraint in the prompt
Notice where the money went. Not into writing code. Into proving the code.
Two moments worth the screenshots
At $47.22 on the client display, the session hit our internal $50 credit cap and stopped cold: the gateway rejected the next call on insufficient pre-authorised quota, with $0.052 to spare. We raised the cap, and it resumed mid-thought without losing the thread.
Then the context window nearly ran out: 215.9k tokens against a 200k budget, the kind of overflow that usually ends an agent session in confusion. It compressed its own working context down to 30.3k and kept going, losing only scrap notes. Watching an agent manage its own memory pressure is something we had read about. Watching it at 3 a.m. with a meter running is something else.
The verification rig it invented
This is the part we keep thinking about. Nobody asked for most of it:
- Equivalence differ: mock the external dependencies so a deterministic hash can act as judge, then run old and new through 8 seeded scenarios and compare every file, every log line, every exit code. 8/8 identical.
- Coverage self-check: it counted branch hits in its own test runs to make sure both versions actually executed the paths being compared, guarding against the false equivalence of "neither side reached that code."
- Static checks: a small undefined-name linter (one benign false positive: it flagged
__file__), plus 13 + 4 unit tests, all green. - Honest changelog: a CHANGES file listing every behavioral difference, including branches it did not cover, plus five judgement calls it explicitly refused to make alone.
We re-ran the full acceptance suite independently, with a different model acting as reviewer. All green.
Was it worth it?
First, a correction we owe the numbers, because it changes how to read them. The $51.77 above is a bill from one specific gateway configuration, not standard Opus pricing: our gateway was stripping the client's reasoning-effort parameter at the time, forcing the model to run at maximum reasoning depth for the whole session. We call this internal artifact a throttle tax. The K3 figure is likewise an extrapolation from a different task profile, not a head-to-head rerun. We plan a proper matched comparison; until then, treat the gap as directional.
As a combined capability probe and critical refactor: yes. The deliverable was not just code but an equivalence test suite and a permanent regression asset; the tests outlive the session.
As a daily working mode: no. Kimi K3 does this class of task at about half the price and roughly twice the speed in our experience, and for routine work the gap in output quality does not justify the gap in bill. The honest answer to "when is full throttle worth it" turns out to be narrow: when the cost of a subtle behavioral regression dwarfs the cost of the session.
What actually changed for us
The refactor shipped, with two small fixes we made ourselves (an empty-string dedup rule and a partial-failure counter). But the durable asset is methodological: machine-checked equivalence as the price of admission for AI-driven rewrites. We have since started folding that pattern, differ plus coverage check plus honest changelog, into our own engineering rules.
An agent that spends $9 of a $52 budget building the apparatus to prove itself right is doing something most human teams skip. That, more than the 1,570 lines, is what we paid to see.
These numbers are field observations from our own gateway and configuration, metered like customer traffic. An outside reader cannot reproduce them as-is, and we would not trust anyone who claimed to.
The series, in order
- Four reviewers, one spec, six agreements: convergence is signal, not validation
- The $10 test harness: the reviewer that built what it recommended
- The $51.77 refactor (you are here)
Meter every session the same way: any OpenAI-compatible key, your tool of choice, an itemised receipt per call. $0.50 free credit; top-ups start at $20