Before we ran the experiment, we wrote down the rules that would kill it. Four predictions and one kill criterion, committed to disk before a single skill was generated: if our structured skill-forge pipeline could not measurably beat a model writing its own skill file in one shot, we would say so, in public, and change the product.
Then the results came in. One-shot prompting beat the forge 98.3% to 91.7% on trigger recall, with overlapping Wilson intervals. By our own pre-registered rules, the hypothesis was dead.
This is a report about what a failed experiment is good for. The loss forced three findings we were not looking for, including a maintenance instruction that scored 0-for-20 until we moved it three paragraphs, and rewrote what our tool claims to do.
This is part 1 of the Skill Forge series: three posts from one experiment program, best read in order. Part 2 is the map of where skills help, do nothing, and backfire; part 3 is the 6KB size law. Each one also stands alone.
The tool, in thirty seconds
Our skill-forge pipeline builds skill files for AI agents: structured instruction documents that say when to trigger, what to do, what never to do, and what to record afterward. The pipeline works in two stages (an analyzer that interviews you about the task and derives a skeleton, then a generator that produces the skill from that skeleton) and then keeps the file alive afterward: a mechanical audit, a failure ledger, and controlled revisions. The hypothesis under test was the obvious one: skills built this way should outperform skills the model improvises for itself.
The experiment that killed it
The P3 round was a routing test with the weak model we use for volume work, DeepSeek V4.1-Flash. Ten task needs, twelve trigger questions each, two trials, both arms: 480 routing judgments, with the on-screen order shuffled by seed so position could not favor either arm. The forge arm ran the full pipeline; the one-shot arm just asked the model to write the skill file directly.
One detail kept the loss from being a shrug. Every forge-arm miss failed the same way: the model declined to trigger at all. It never picked the wrong skill. Something in our generated files was telling the model to stand down.
Finding one: boundary sentences collapse recall
The culprit was the description field. Our generator had been writing boundary clauses into it, "does not trigger for X" and "only when Y", the kind of negative scoping that reads like good hygiene to a human. To a small model deciding whether this skill is relevant, those clauses were stop signs. We patched the generator (v0.3.1) to keep boundary sentences out of descriptions entirely, and widened the trigger-phrase window from 20 to 60 characters. On the re-run, misses from this failure mode went from 10 to zero.
A second round, P4, moved the test to harder ground: eight needs drawn from one real project domain, where every skill has near-neighbors and the trap questions sound like they belong to two skills at once. The kill rule fired a second time, but the recall gap narrowed from 6.6 points to 2.7, false-trigger rates leveled between the arms, and the failure mode changed shape. Instead of refusing to trigger, the model now picked the wrong neighbor, with one of our internal ops skills acting as a gravity well. Both arms fell for the same trap question twice out of two; the hand-tuned champion version of that skill answered it perfectly, because its description carried a routing verdict naming exactly that confusion. No generator saw its neighbors, so no generator could write that sentence. That became v0.3.2: the analyzer now asks you, at generation time, which sibling skills yours must not be confused with.
One more thing survived both losses. The mechanical audit, the boring structural health check, caught the one-shot arm inventing a pitfall that never happened and quoting a price that does not exist. The forge arm: zero red flags, zero inventions, across every generated file. Discipline was doing something, just not the something we had predicted.
Finding two: 0 for 20
The most valuable result came from the smallest design detail. Every forge skill ends with a maintenance instruction: after running, if something went wrong or is unfinished, record it to the failure ledger. It sat at the bottom of the file as a footnote, the way "remember to update the docs" sits at the bottom of every README ever written. The actual footnote, translated from our internal files:
If this skill fails or misfires during a run, and a skill_feedback tool exists in the environment, call it with the skill name, the scenario, and one line about the failure. If the tool does not exist, skip silently; never invent the call.
Across twenty executions on two models, the instruction fired zero times. Not poorly. Never.
The fix was not better wording. We had already tried that: standalone section, single-line note, no difference. The fix was moving the instruction into the numbered task steps, as the final step of the work itself:
Closing step: review this run for anything abnormal, unfinished, or failed. If so, call skill_feedback. If not, pass silently. Same sentence, new address. The weak model's feedback score went from 0.00 to 1.80 out of 2.
The strong model added a nuance we did not expect: it sometimes logged feedback with no instruction at all (1.33 out of 2, unprompted), and the embedded step only lifted it to 0.67 on a small sample. The lesson generalizes anyway, and it generalizes past our tool: an agent does what the task steps say, not what the footnotes hope. If your AGENTS.md ends with "remember to update the changelog," you have a footnote. We sent this experiment file to an outside reviewer who had been harsh on everything else; this was the finding they called the most valuable product insight in the document.
What we changed the claim to
Two falsified rounds plus one structural win forced a rewrite of the product's own description. The old claim that structured generation improves skill quality, full stop, is dead, and it died the way we wanted it to: on a rule we committed to before knowing the answer.
What survives is narrower and more useful. The forge does not make models equally capable; it makes the quality requirements consistent across models: the same triggers, red lines, acceptance checks, and failure records, whoever executes. It raises the floor. It does not promise the ceiling. And its center of gravity moved from authoring to lifecycle: the audit, the ledger, the controlled revision. Generating a skill is a moment. Maintaining one is the job.
There is a coda we did not plan. A later round pre-registered the "ideal form" hypothesis, a task class where both models start mediocre and the forge lifts both, and that hypothesis failed its pre-registration too, three predictions out of four missed. The same round produced the program's most-quoted discovery almost in passing: a naturally bloated 9.2KB skill file drove the weak model to empty responses six times in a row, which we then isolated into a clean size-gradients law. We wrote that one up separately: the 6KB line.
What this cost, and what it cannot tell you
The experiments in this piece (P3, P4, the execution-layer rounds, the pre-registered v3 round) cost ¥10.6 in API fees, about $1.50. Falsification is the cheapest R&D there is, if you write the kill rule down first.
The limits are real and we will save you the trouble of listing them against us: small samples (n=5–10 per arm in the execution rounds), one model family, a judge model related to the weak executor, simulated rather than real tool execution, and single-pass generation with its own variance. Where a difference was small, we treated it as noise, and the Wilson intervals are on the chart, not in a footnote. The failure record is the point of the piece; it would be strange to launder it.
The series, in order
- The kill rule that fired (you are here)
- Same skill, opposite effects: where the forge actually pays, mapped across weak and strong models
- The 6KB line: the size law from the coda above
Beyond the series: SkillRefiner: five stages, zero replays, the paper that runs a loop like ours, and what it taught us back. And after this report shipped, we paid Claude $10 to review the forge spec; it sent back a test harness.