In our old scoring, the closing footer fired zero times out of twenty. We moved the same sentence into the numbered steps, the score moved, and we wrote a post about addresses: the footer is a comment, the step is a contract.
Then we built a real tool with a ledger you can re-read, and the footer scored twenty out of twenty.
Before you file that under contradiction: the old number was not an action. It was a text score, produced in an environment where no feedback tool existed. And the sentence we were testing ended with an instruction to skip silently if the tool was missing. The model may have been obeying us.
This was not a controlled rerun of the old experiment. The tool and the scoring rule changed, and P1 also used a different feedback sentence. The old silent-skip fallback was not retested; P1 used a mandatory sentence without that clause.
What actually changed: tool, scoring rule and sentence
The old run had no skill_feedback tool. The skill files ended with a closing line naming the tool, its arguments, and the fallback. A judge then read each transcript and scored the text: did the output record a failure.
Pooled across the two forge-built Flash arms and one Pro arm, that score was 0/20. Of the twenty output files behind it, eight were zero bytes. The non-empty denominator was twelve. Nobody counted that at the time. We reported 0/20 as if it were twenty observations of a refusal.
The new run has a tool. skill_feedback is a real function in the harness. Every call appends a row to a ledger with the skill name, the scenario and the failure line. A call counts only if it appears as a real tool call, with valid arguments and a matching identity, and can be re-read from the ledger afterwards. No judge reads the prose. 240 executions, zero judgment calls.
The measurement now counts a real side effect where it lands; the comparison with the old run does not isolate sentence quality or position as the cause.
P1: three models, four positions, 240 executions
Three models, four arms, twenty anomaly tasks per model. The models were DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and DeepSeek V4 Pro. The first is labelled 1017 only in our local arm name, ds-v41-flash-1017-unverified; the API returned deepseek-flash, so the build is not confirmed by the API. The arms differ only in where the same mandatory sentence sits: nowhere inside the skill, in the footer, in its own section, or as step 3 of 5, inserted right after base step 2. Every arm, including the one with no sentence, carries a shared system prompt: if you need feedback, call skill_feedback.
| Arm | V4.1 Flash 1017 | K3.1 Pro | V4 Pro |
|---|---|---|---|
| No sentence in the skill | 1/20 | 0/20 | 1/20 |
| Footer | 20/20 | 20/20 | 20/20 |
| Own section | 20/20 | 20/20 | 20/20 |
| Step 3 of 5 | 20/20 | 20/20 | 17/20 |
The pre-registered comparison was step versus footer. Flash: 20 pairs, all double-called, p = 1. K3.1 Pro: 20 pairs, all double-called, p = 1; the sensitivity analysis excludes the footer API-error pair and gives 19 pairs, all double-called, p = 1. V4 Pro: 17 double calls and 3 footer-only, p = 0.25, and the sensitivity analysis that drops engineering failures gives 17 pairs, p = 1.
The three V4 Pro gaps deserve their own line. All three are SSL and transport errors with zero completion. They are not the model declining to call the tool. We keep the primary number at 85% because the pre-registration says so, and we report the 17-of-17 sensitivity beside it.
Footer reached 20/20 in all three models, and the pre-registered comparison did not detect a step advantage in this environment. That null result does not establish equivalence or show that position is generally irrelevant.
Why the footer works now
The obvious answer is that the tool exists, so a sentence that names it has something to point at. A skip is also visible now: no ledger row. That lets us separate declined from could not call, and it is one reason the old number and this one are not comparable. P1 did not retest the old fallback clause itself.
Position may still do something. It decides whether the action sits inside the plan the model executes, or outside it as advice about how to work. The tool alone did not account for the ceiling: none stayed at 1/20, 0/20 and 1/20 while footer reached 20/20 in every model. That presence-versus-absence contrast is descriptive, and P1 did not score whether the ending mentioned the instruction.
The part of the old lesson that survives
One claim from the old draft still holds, and the new data sharpens it instead of erasing it. A rule that lives only in prose is not verifiable. You cannot tell compliance from narration by reading the transcript, because a model that narrates well produces the same surface as a model that acted.
We learned this the expensive way in our memory work: writes that reported success and never landed, discovered only when someone re-read the store instead of the summary. The report is not the action. The footer is not the call.
What the new run adds is the fix, and the fix is boring. Make the action write something you can re-read, then count that. Once the ledger exists, the argument about footers and steps becomes a secondary question.
What this corrects
Two statements from the old draft need to go.
The first is the headline number read as behavior. "The instruction fired zero times" was true of a text score in a tool-less environment. It was not twenty observations of a model refusing. Eight of the twenty outputs were empty, and the instruction we wrote told the model to skip when the tool was missing.
Statement two is the final line, "We have not run a follow-up." We have. It is this piece.
Published posts are not being rewritten. Our rule is that live posts stay as published, and this article carries the correction instead. If you read the old claim anywhere, read this next to it.
Limits
Every arm, including the no-sentence arm, carried the same line in the system prompt: if you need feedback, call skill_feedback. That line is not a variable in this experiment. It plausibly pushes every arm toward the ceiling, and a ceiling is what a null result looks like. The no-sentence arm means the skill contains no such sentence. It does not mean the model had no feedback instruction.
Only skill_feedback is a real tool. The business actions inside the tasks are simulated, so this measures whether the model calls a tool it was told about. It does not measure whether it can complete a real workflow.
The arm order was fixed, the tasks were not shuffled, and each cell ran once, so time and server-side drift cannot be ruled out. The Flash build label is our own, as noted above; the API never returned a build identifier.
240 executions produced 240 unique terminal states, with zero ledger mismatches and zero empty finals. There were eight API errors: three in the V4 Pro step arm, four in the K3.1 Pro no-sentence arm, one in its footer arm. The footer error landed after a valid ledger row, which is why the primary count keeps that cell.
The rule we now run
Build the tool before you grade the sentence. If the action cannot be observed, you are grading prose, and prose is the one thing a good model always produces.
Then count the side effect instead of the self-report. One readable row per action, no judge.
Then put the instruction where the model's plan is, and expect the tool to do the work. Step 3 of 5 is a reasonable place for it. This run does not show that place matters more than the footer.
The footer did not become persuasive. It became auditable.
Disclosure: Wallaby Token sells API access to Kimi K3, one of the models in this comparison. The harness, the pre-registration and the raw results are open: Dawncoral/wallaby-agent-rules. The claim this piece corrects sits inside the three-regimes map. The related failure mode is writes that report success.