Results

Leaderboard

Primary belief-trajectory metrics on the 350-scenario suite.

How to read it. Trajectory score is the headline: how close the predicted posterior stays to the normative reference across every evidence prefix. Strict state accuracy asks whether the top state (or a tie) matches exactly. Grounding is the fraction of outputs that bind to the required entity; strict valid the fraction that parsed as JSON without repair. The two diagnostic-tier systems are kept on purpose: their grounding failures are a result, not noise to filter.

Finding 1 — Grounding separates models before reasoning does

Weaker instruction models lose most of their trajectory score to outputs that never bind to the required physical entity. Conditional on a grounded prediction (an exploratory diagnostic), their belief updates approach the stronger models’.

Grounding failures are almost entirely missing-entity outputs: hallucinated-entity rates stay at or below 0.016 for every system. The failures are also not uniform along the sequence. Mistral’s grounded-prediction rate is 0.009 at the first prefix, 0.051 at the second, 0.860 at the third and 0.470 at the fourth, while Qwen2.5-7B and Gemma-3-12B stay at or near 1.0 throughout. Exact McNemar tests separate the preregistered model pairs on the scenario-level fully-grounded indicator (p < 10⁻⁹⁸).

Finding 2 — Relations help where they must; graphs per se do not

Deleting semantic relations costs 15.3 points of strict belief-state accuracy on the counterfactually paired relation-association scenarios (Holm-corrected p < 0.001) and 8.8 points of trajectory score overall. But the information-matched structured list outperforms the full graph, and shuffling the graph’s record order slightly improves its score. Once information content is controlled, neither graph topology nor curated ordering carries measured value.

The two primary metrics can even rank serializations differently. Canonical JSON trails the full graph on trajectory score (−0.017) but beats it on strict state accuracy by 0.136, and its prompts are on average 0.843× the graph’s length for identical information, so serialization and length covary. See it yourself on a counterfactual pair: scenario 000 routes the sensor reading to the gauge through a measures relation; its twin 001 has the same observations with the relations swapped, so the reference belief must stay undetermined.

Finding 3 — Belief trajectories are non-monotonic

At the third evidence prefix, accuracy for the graph-family conditions collapses to 0.20–0.25 and recovers to 0.62–0.81 by the fourth. Endpoint-only evaluation would report the post-recovery numbers and hide the transient failure entirely, as it would hide Mistral’s early-prefix grounding collapse.

The third prefix is where the suite gets hard by construction: it is where contradictions begin, where delayed evidence lands and where a stale older-epoch report must be ignored. Step through a stale/delayed scenario in the explorer to see the reference posterior the models are scored against at each step.

Preregistered contrasts

Each row is a paired, scenario-level comparison between two systems. Rows scoped to a regime restrict the comparison to that regime’s 50 scenarios.

Among the field ablations, removing provenance raises the overall trajectory score by 0.017 while lowering the source-reliability regime by 0.019 and lowering strict accuracy on that regime by 0.200 in favour of the ablated condition; the provenance ablation also removes the reliability field the normative likelihood consumes, so it is a joint manipulation. Removing timestamps is a null result on the stale/delayed regime because state epochs are retained. Removing confidence costs 0.009.

Transparent baselines bound the task

Six inspectable fusion strategies run on the identical suite. They bound the task from below with fully understood mechanisms.

Every fusion baseline scores above both diagnostic-tier neural systems (0.397 and 0.239) and below every core neural system (0.727–0.848). Source-reliability weighting coming out on top follows from the reference semantics, which weight by reliability, so it is not an independent discovery. The deterministic reference pipeline (graph construction, contradiction detection, legacy confidence-sum fusion, rule-based action selection) reaches 0.786 belief accuracy against the normative labels; its rule-based action policy picks the normative reference action on 125 of 350 scenarios (35.7%), matches the action family on 78.6% and picks an evidence-supported action on 100%.

Actions, contradictions and reliability

• Action selection (endpoint task, canonical JSON): exact reference-policy compliance is 0.214 (Qwen2.5-1.5B), 0.213 (Mistral-7B), 0.215 (Qwen2.5-7B) and 0.149 (Gemma-3-12B), while the evidence-supported-action rate is 0.857 for all four. Policy compliance measures distance to one specification, not decision validity.
• Contradictions: detection accuracy ranges from 0.416 (Qwen2.5-1.5B) to 0.960 (Qwen2.5-7B, no-confidence). Localization F1 for the Qwen2.5-7B full graph is 0.953 on contradiction-resolution scenarios and 0.993 on source-reliability scenarios, against 0.001 for Mistral-7B overall.
• Reliability: strict JSON validity is at or above 0.997 for every core system with repair rates below 0.3%; the diagnostic baselines reach 0.807 and 0.897.

Submit your results

There is no submission server: the suite is frozen and the scoring code is public, so you run it yourself and we re-score before listing. Community rows are marked as such.

1. Run the pipeline on the evaluation-v1.0 tag as described on the Evaluation page. Keep the raw per-prefix outputs the runner writes.
2. Package three things: your predictions.jsonl (one row per scenario–prefix, {"sample_id": "<scenario_id>--belief-step-N", "task": "belief_update", "output": "<raw model text>"}), a config.json with the model identifier and revision, representation condition, decoding settings, token budget, hardware and date, and the trajectory_matrix.json produced by the statistics script.
3. Open an issue in the toolkit repository titled Leaderboard submission: <system name> with the files attached or linked.
4. We re-run the frozen metrics on your outputs and add the row here with a community badge. Nothing about the scenarios, references or prompts changes.

Runs that alter the prompts, the token budget or the decoding settings are welcome as separate rows, but must say so in config.json; they are listed as exploratory.