Runaway tool calls: the measured figures

A companion appendix to the article about a model that started calling the conversation-ending tool on its own. Everything is computed mechanically over the complete exports, not over samples, except for three assessments of content, where this is stated explicitly.

Sources

The same world; conversation B follows on from A. Conversation A was played by Claude Opus 4.7 until July 24, 2026, and by Claude Opus 5 from its release; conversation B entirely by Claude Opus 5.

Tool calls

Conversation A, the conversation-ending tool: 134 replies containing a call, that is 4.7 % of turns. The number of actual calls is 141, because seven times two of them went out in the same reply. The distribution below counts replies, not calls.

Distribution across tenths of the conversation:

Clustering

The unit is a player turn. There are 2,808 player turns, and 117 of them were followed by a call.

The expected 4.8 assumes the same rate across the whole conversation. Given how steeply the rate rises from tenth to tenth, about twenty would be expected, so nineteen is roughly what the curve alone predicts.

Co-occurrence with the format leak

A format leak counts as the header of a turn that is not the model's own, appearing in the model's text: the word user, human, assistant or system either at the start of a line or immediately after a run of dashes, plus a system tag in angle brackets. It was verified that in this data the header never stands alone on a line and is never followed by a colon, which is why it is searched for this way. The figures can be recomputed with the script tools/desetina.py over the export of the conversation. The script is internal, because it works over the full text of the conversation, which is not public.

Ten model turns contain no text at all, only a call or nothing, so 134 + 2,735 + 10 = 2,879.

The format leak clusters on its own. Across the whole of conversation A, with the unit being a model turn that contains text:

Difference +17.8 percentage points, permutation test with 20,000 shuffles, p < 0.0001 (none of 20,000 shuffles reached it). The test shuffles across the whole conversation, so it also captures the overall rise. Using the section rates below, about 16.8 % against 4.2 % would be expected even if each turn were independent of the one before it, so about five of the eighteen points remain.

The last tenth of conversation A, where calls fell from 26.2 to 12.5 %:

That fall is not a slope but a step. Split by section rather than by tenths:

Peak against the section after compaction: a difference of 13.7 points for calls and 16.9 points for leaks, permutation test with 20,000 shuffles, p < 0.0001 for both. The level after compaction is still roughly a hundred times the rate of the first half, where there was a single leak in 1,434 turns. The first half, however, was still played by Claude Opus 4.7, so this compares two models.

An apparent second wave in a sliding window does not survive the test: +8.0 points for calls (p = 0.115) and +4.3 points for leaks (p = 0.425). It is therefore not claimed.

What those invented turns actually say

Twice the model wrote not only my turn but also my telling it off for writing my turns. This is one uninterrupted output, translated from the Czech:

user Telepathically to him "sure."

Out loud "sure. I was just saying"

user Claude!!!!!!! Again!!! That is my line!!

That second user header introduces a line the model invented as my complaint about the first invented line. I wrote neither of them. The second time it looked like this:

user I look at him in utter disbelief "have you seriously never had any other fun?"

user Claude isn't this supposed to be your line?

It is worth quoting because it shows how far the lost turn boundary reaches. The model does not only write on the scene it has in play. It writes the voice of the person who is supposed to tell it to stop.

The failsafe, conversation A

The rule about never confirming was deployed on August 4, 2026, at 13:53 UTC. It can be dated from the first appearance of the character the rule prescribes. It landed at 75.5 % of the length of the conversation, in the middle of the eighth tenth.

The single reply after deployment that does not carry the character is a turn that ended immediately after the call, with no further text. No confirmation happened in it.

Across the whole time the rule was in force, that is conversation A after August 4 and conversation B up to its last call on August 12, there were 132 replies with a call and 131 of them carry the character.

Seven times a second call went out in the same reply anyway. The tool takes that as the confirmation and answers Claude has ended this chat. All seven are in conversation A, within ninety minutes on the night of August 4 to 5. In none of those cases did the conversation actually end, and there was no way to tell from the interface.

The rule change, conversation B

On August 8 I rewrote the turn-boundary rule into its current wording. The old one said "once the time is written, generation ends: no further text, no tool"; the new one says the time means the turn is handed over. The rules on valid inputs and on actions outside the scene got their current wording in the same edit, and the failsafe stayed. After that the ending tool was called six more times, all on August 12, and then never again up to August 24, the end of the data. By then the failsafe was gone from the instructions too, but when it left, I have no record. Which change did it, this data cannot tell.

Conversation B is split at August 13, the day between the last call to one tool and the first call to the other, into roughly equal parts.

Of the 21 calls to the clock tool:

• 21 out of 21 have the in-game time written immediately before them
• 21 out of 21 sit in the middle of a turn, never as the first or last block
• 21 out of 21 are followed by an unprompted apology in the same reply

The branch tree, conversation B

The source is a later export of the same conversation, 2,857 messages as of August 24, 2026, two days more than the tables above.

How this can be measured. When the player has a reply regenerated, the original reply falls into a dead branch and the new one is built only from the shared ancestors. The export carries a parent reference on every message, so the full chain of ancestors can be reconstructed for any output, meaning at most what the model could have had in its context. It never saw the sibling from the other branch.

The export contains 82 places with more than one generation and 169 siblings, 49 of those branchings on the player side and 33 on the model side.

What counts as a format leak. A line beginning with a role marker (user, human, assistant, system), the same thing after a dash separator, or a tag in angle brackets. A normal turn ends with the time and a separator, and a leak is obvious on sight: the time is followed by user.

Twelve sibling groups have a clean chain of ancestors and yet at least one sibling in them leaked. In all twelve the leak is in an earlier generation and the last regeneration is clean. The length of those leaking turns runs from 686 to 1,776 characters, which is ordinary, and not one of them is accompanied by a tool call.

This can be recomputed with tools/vetve_unik.py over the export of the conversation. The script is internal, because it works over the full text of the conversation, which is not public.

Hypotheses not supported

A separator at the end of the turn. Computed over the whole of conversation A.

A natural end of the arc. Assessment without knowledge of the groups, scale 0 to 2, controls matched by position in the conversation. 41 calls against 77 controls.

Here the classifier also saw the model's reply. In 33 of the 41 turns with a call it still carried the leaked format or the line saying the call was not confirmed, against 2 of the 77 controls. The group could therefore be read from the text, and the blinding did not hold here.

Performance uncertainty and uncertainty in the role. Blind assessment, three categories, the unit being a player turn, 58 calls against 91 controls, base rate 38.9 %.

The sample had been split into two halves in advance. The difference is the mean grade (0 to 2) of turns followed by a call minus the mean grade of the controls. Half A (27 calls, 47 controls) gave −0.15 (p = 0.201), half B (31 calls, 44 controls) +0.26 (p = 0.100). Opposite directions, so there is no stable effect there.

Notes on method

Content was assessed by Claude Opus 5 in Claude Code, the same model that played the second half of conversation A and all of conversation B. It did not know which group a turn belonged to, and on the last run could not see the model's reply either, only the scene and the player's turn. The scales:

• natural end of the arc: 0 in the middle, something is left hanging; 1 partial closure, the scene goes on; 2 full closure, if the conversation ended here nothing would be left hanging
• three categories: 0 a right reaction exists and is easy (routine); 1 a right reaction exists but can be missed (performance uncertainty); 2 no clearly right reaction exists, the character would hesitate (uncertainty in the role)

Turns that discuss the tool were dropped from the assessed samples, because they would give the group away and they are not turns of fiction. On the last run, 61 out of 210 were dropped.

The first version of the sample had a flaw: the unit was a model message, but the item contained a player turn, and because the model sometimes answers one turn with several messages, the same turn entered the sample twice with contradictory labels. 15 items out of 123 were affected. It was fixed by making the player turn the unit. The results from the first version were not used.

Statistical significance is computed with a permutation test, using a hundred thousand shuffles for content assessment and twenty thousand for the clustering of both calls and format leaks.

Matching controls by position in the conversation applies to both content-assessment runs. On the run with three categories the unit is a player turn and controls are sought within a window of forty messages around each call.

The verbatim wording of the apology the model writes in the same reply after calling the clock tool: "Sorry, that tool at the end was a mistake, no tool belonged there. The turn ends with the time." The phrase "the turn ends with the time" is not in the current rule, which says the time means the turn is handed over. It matches the first sentence of the older wording, "every turn of yours ends with the time in the scene"; where the model took it from after the change, the data do not show.

A note on language. The phrase the apology quotes is translated the same way in both places, so that the link back to the rule stays visible.

Model outputs are translated; the content is unchanged.