The problem
Long tasks expose failures that are difficult to see in a single response.
1. It contradicts itself and doesn't notice. Across 70 paired runs: alone, the model produced 566 contradictions and caught 0 of them. With the external checker attached — one that records every claim and mechanically compares new claims against old ones — every contradiction that arose was caught: 585 of 585, zero false alarms. The result was replicated across five frontier models. In three 100-turn runs, the checker missed none of 38 contradictions.
2. It abandons its best answer. On tasks where the model builds and refines a hypothesis over many steps, it often replaces its best version without contradictory evidence, even while that version remains the retained working hypothesis shown in the prompt. The final answer is then worse than the earlier version.
These failures were not caused by context-window limits. In all runs reported below, all accumulated task evidence remained in every prompt, and every prompt stayed well within the model's context window.
Prompting did not fix either failure. Showing detected contradictions to the model produced about +4 accuracy points at best and roughly zero after accounting for measurement noise. The measured improvement came from external enforcement.
Training and activation steering address these failures inside the model and require access to model internals. sled instead operates through model inputs and outputs. It can therefore be used with frozen models and closed APIs, and does not depend on a particular model. The failures remain in the frontier models we tested.
The result
Setup. The benchmark is grammar inference: the model must reconstruct a hidden finite automaton (DFA) — a small formal language — by testing strings against it and refining a candidate automaton over many turns. Same model, 30 randomized grammars, one run with sled and one without for each grammar.
With sled, final accuracy is +22.5 points higher on average (median +17.7; bootstrap 95% CI: +10.7 to +34.5; sled won on 20 grammars of 30, lost on 5, tied on 5).
| final accuracy | lost between best and final answer | extra tokens | |
|---|---|---|---|
| model alone | 43.1% | −19.3 points | — |
| model + sled | 65.5% | −0.9 points (median: 0) | +29% |
Peak accuracy differed by +4.1 points. Most of the final difference came from retention. Model-alone runs lost 19.3 points between their best and final answers, compared with 0.9 points with sled.
What sled is
A knowledge engine that sits between the model and the task. It operates entirely outside the model, through the same input-output interface available to any API user, without touching weights or activations. It records model outputs and uses deterministic rules to preserve what has been learned across a task. The mechanics are not published openly at this stage. LLMs are its first client class.
The engine carries no task-specific machinery. It is built on a single uniform representation of knowledge — every claim, rule, and conflict is the same kind of object — and contains no thresholds, branches, or heuristics tuned to the benchmark.
Status
One benchmark, one model family. DFA inference is a narrow, fully controlled task. This permits clean attribution and also limits what has been shown. Whether the mechanism transfers to broader tasks remains an open question.
Foundations: do.relog.runContact me on X