The problem#
Long-task failures are not new. The runs below measure two specific forms that are difficult to see in a single response.
1. It contradicts itself and doesn't notice. Across 70 paired MiniMax M3 runs on a formal grammar-inference task, 566 contradictions arose in the unassisted runs. The model never flagged or revised any of its own prior conflicting claims: 0 of 566. In the runs with sled's external checker, 585 contradictions arose. The counts differ because the two arms are separate runs. The checker — which records each claim and mechanically compares new claims against old ones — detected all 585, with zero false alarms. The same contradiction behavior was also observed in checks with GPT-5.5, Qwen, DeepSeek, Kimi, GPT-OSS, and Nemotron models. Those checks are not included in the figures reported here. In three 100-turn MiniMax M3 runs, the checker detected all 38 contradictions.
2. It abandons its best answer. On tasks where the model builds and refines a hypothesis over many steps, it often replaces its best version without contradictory evidence, even while that version remains the retained working hypothesis shown in the prompt. The final answer is then worse than the earlier version.
These failures were not caused by context-window limits. In all runs reported below, all accumulated task evidence remained in every prompt, and every prompt stayed well within the model's context window.
Showing detected contradictions to the model produced about +4 accuracy points at best and roughly zero after accounting for measurement noise. In the benchmark, the larger improvement came from external enforcement.
The result#
Setup. The benchmark is grammar inference: the model must reconstruct a hidden finite automaton (DFA) — a small formal language — by testing strings against it and refining a candidate automaton over many turns. MiniMax M3 was tested on 30 randomized grammars, with one run using sled and one without it for each grammar.
Accuracy is the fraction of all 2,047 strings over {a,b} of length at most 10 (including the empty string) on which the proposed and hidden automata agree on membership. Peak and final accuracy are computed after the run by an evaluator with access to the hidden automaton. Neither score is available to sled during the run. Final accuracy scores the last proposed automaton in model-alone runs and the hypothesis retained by sled in sled runs. Peak accuracy is the highest such score reached during the run.
With sled, final accuracy is +22.5 points higher on average (median +17.7; bootstrap 95% CI: +10.7 to +34.5; sled won on 20 grammars of 30, lost on 5, tied on 5).
| peak accuracy | final accuracy | lost from peak | extra tokens | |
|---|---|---|---|---|
| model alone | 62.3% | 43.1% | −19.3 points | — |
| model + sled | 66.5% | 65.5% | −0.9 points (median: 0) | +29% |
Peak accuracy differed by 4.1 points. sled's measured effect was mainly retention: preserving a better hypothesis already found rather than substantially increasing the best accuracy reached. Model-alone runs lost 19.3 points between peak and final accuracy, compared with 0.9 points with sled.
What sled is#
sled is an external knowledge engine between the model and the task. It records claims and conflicts, maintains a current working hypothesis, and updates it from evidence returned by the task. The model itself remains unchanged.
Status#
One benchmark, one model family. DFA inference is a narrow, controlled task. Whether the mechanism transfers to broader tasks remains an open question.