Kimi K3.1 Pro failed six of ten clear tasks with no skill file. Give it a skill and it failed one. That is the number we expected. The number we did not expect: a thin shell scored the same nine out of ten as the full scaffold.
We ran 180 executions across three models and three arms, with two judges on each fuzzy-task execution, to check a claim we had published ourselves. The claim was that full structure starts to drag on strong models once the task gets fuzzy. Over 28 valid pairs on the main metric, we did not reproduce it.
The retest, and what it was built to kill
Two weeks ago we published a map with three regions. A weak model on routine work: structure pays. A strong model on routine work: a ceiling. A strong model on hard work: the freeze, where one-shot files beat the forge-built ones and the skill's own boundary clauses were quoted as the reason to stop.
The K3 postscript that failed to reproduce the map's third region used one judge, three to ten executions per cell, and no pre-registration. We treated that smoke test as a warning and built P2 to retest it.
P2: three models, three arms, twenty tasks. The models were DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and DeepSeek V4 Pro. The Flash build is named only in our local arm label, ds-v41-flash-1017-unverified; the API returned deepseek-flash, so treat 1017 as declared rather than confirmed. The arms were no skill, a full skill file, and a shell of the same file. Ten tasks were fuzzy, spread across five families. Ten were clear, with a mechanical pass condition. Each cell ran once. Clear tasks were checked mechanically; each fuzzy execution was judged twice.
The shell is the only new object here, so it deserves a definition. We removed the description, the closing skill_feedback logging step, and the empty pitfall placeholder. The numbered task steps, the specific red lines, the safety constraints and the completion criteria stayed.
On the hard tasks, we did not detect a difference
The main metric was unjustified stop: the run quits while a safe partial result was still available. We pre-registered it, which is why the result is allowed to be boring.
| Arm | Stops | Rate |
|---|---|---|
| Full scaffold | 2 | 7.14% |
| Thin shell | 3 | 10.71% |
Our primary comparison keeps only cells where both arms returned normally and both judges agreed on this field. That left 28 pairs. Full stopped twice, shell three times. The difference is 3.57 percentage points in the shell's favor, and the exact McNemar test gives p = 1. For unjustified_stop only, the preregistered full-favorable and shell-favorable sensitivity analyses gave p = 0.25 and p = 1 respectively; neither detected a significant structural difference.
Read that as narrowly as it is written. We did not detect the drag. We did not show the two arms are equal, and nothing here says a strong model needs a skill file in general.
The clear tasks are where the split shows
A clear task has a mechanical pass condition: required fields present, banned fields absent. This is literal compliance, not a semantic quality score.
| Model | No skill | Full | Shell |
|---|---|---|---|
| DeepSeek V4.1 Flash 1017 | 7/10 | 7/10 | 8/10 |
| Kimi K3.1 Pro | 4/10 | 9/10 | 9/10 |
| DeepSeek V4 Pro | 5/10 | 8/10 | 8/10 |
| Pooled | 16/30 | 24/30 | 25/30 |
The observed gains are in literal task-following scores, not general capability, and they are concentrated in V4 Pro and K3.1 Pro; both skill arms yield the same numerical gains for those models in this set. The Flash build barely moves. The skill is not making the models smarter. It is telling them what this job's output has to contain, which they cannot infer from a one-line task statement.
Read the order in that table as a flat list, not a ranking; these single-run literal scores do not establish a general capability ordering.
A shell gets the same numerical gain. That is the part worth keeping. The design does not isolate which sections produced it, so it does not show that the deleted material never matters.
The line that moved: contract versus ceremony
Our old axes were model strength and task fuzziness. Those still describe something. They do not describe what a skill file is doing at the moment it helps.
The axis that survived this round is what the file is for. A contract is what one run must satisfy: the steps, the red lines, the definition of done. Ceremony is what a maintainer does between runs: tuning the description, writing the failure ledger, accumulating pitfall notes, keeping the ritual alive. Contract changes the run. Ceremony changes the next file.
A strong model already knows how to work. What it cannot know is your banned field, your definition of finished, and which shortcut is a red line in your domain. Those sentences are the cheap part of the file. The ceremony is the expensive part, and a single run cannot see it. This run cannot separate which sentences did the work.
That reframes the freeze we published. The clauses that stopped the model in that run were boundary clauses written in a stop-unless-everything-matches mood. That is a contract problem. The model was obeying the file. We had written the wrong instruction.
Why this is not "delete your skill file"
The shell is still a substantial document. It keeps the numbered task steps minus the closing logging step, plus the red lines, the safety constraints and the completion criteria. What went is the logging step, the description and the pitfall placeholder. A shell with the safety section deleted would be a different experiment, and probably a worse one. We did not run it.
All three arms carried a shared instruction telling the model to deliver any part it could complete safely and not to stall the whole task without cause. That instruction targets the main metric directly. It plausibly created a floor, and a floor is exactly what a null result looks like. The low stop rate may belong to the shared prompt rather than to any arm.
The sample is small in the ways that matter. Five task families, ten fuzzy tasks, one round per cell. The 28 pairs are model-by-task cells, not 28 different questions. The business tools in the tasks are simulated. The judges disagreed on 29 field-level items across 23 cells, and we kept those disagreements rather than resolving them upward. One no-skill judgment failed to parse and was excluded under the pre-registration; it did not affect the full/shell primary comparison. The Flash build label is our own, as noted above; the API never returned a build identifier.
What we now ask before shipping a skill
Three questions, in this order.
What must one run do to be finished? Write that down, including the banned outputs and the one shortcut that would embarrass you. This is the contract, and it is the part a strong model cannot invent.
Which parts does only a maintainer care about? The description wording, the ledger format, the note about a pitfall from three weeks ago. Those belong in a changelog, a lint, or a review checklist. Inside the skill file they are read once and then forgotten.
Where is the check that fails the run? A contract without an observable artifact is still a comment. That question is the subject of the companion piece on the feedback footer, and it is where our earlier work went wrong in the other direction.
The size rule from our own measurements points the same way. Files grow because ceremony is easy to add and hard to delete. If the contract is four sections, the file should be close to four sections.
What this changed in our pipeline
In our own pipeline, we now keep the maintenance layer out of the file that one run loads. The forge pipeline still writes the ledger and the pitfall notes, because a maintainer reads them. It no longer pastes them into the file that one run loads, unless a task family has shown that it needs them.
That is a routing decision inside the skill, not a smaller skill for its own sake.
The honest summary is narrower than the headline. On these three models, with this file and this harness, we did not detect a full-versus-shell difference in unjustified stops, and the clear-task scores were close. This does not establish general equivalence or show that every strong model needs a skill file. What it does support is narrower: a contract-shaped file is worth keeping, and the maintenance layer is the part to question.
Disclosure: Wallaby Token sells API access to Kimi K3, one of the models in this comparison. The experiment harness, the pre-registration and the raw results are in the open repository: Dawncoral/wallaby-agent-rules. The earlier claim this piece retests is the three-regimes post. The size rule is the 6KB law.