Evaluation

CREG scores each output as a separate layer. A system can produce valid JSON but never name the right entity; it can track the posterior well but miss a contradiction; it can pick a defensible action that is not the one the reference policy prescribes. The protocol keeps these apart instead of collapsing them into one accuracy number, and it scores the whole belief trajectory, not just the endpoint.

The layers

Layer	What is scored	Headline metrics	Failure looks like
Reliability	Did the output parse?	strict JSON validity; repair rate	Prose around the JSON, truncated objects, missing fields. Missing or malformed outputs score zero on categorical and F1 metrics and maximal trajectory distance.
Entity grounding	Did the prediction bind to the required physical entity?	required-entity rate; missing-entity and hallucinated-entity rates	A belief about `gauge-2` or a made-up entity when `gauge-1` was asked for.
Belief trajectory	How close is the predicted posterior to the normative reference at every prefix?	trajectory score (primary); strict state accuracy; preservation accuracy; correct-revision rate; Brier distance; update latency	Flip-flopping on weak evidence, failing to reverse on strong evidence, applying stale evidence.
Evidence attribution	Which observations support or contradict the belief?	supporting-evidence F1; counter-evidence F1	Citing the distractor, omitting the correction.
Contradiction handling	Is a same-epoch conflict detected, localized and typed?	detection accuracy; localization F1 (only when a contradiction exists); type accuracy	Reporting a contradiction after it was superseded, or naming the wrong pair.
Action selection	Is the chosen action admissible, in the right family, and supported by evidence?	exact reference-policy compliance; action-family agreement; evidence-supported-action rate	Choosing `ignore`, or an unsafe `shutdown`/`actuate`, or `monitor` during an active contradiction.
Explanation grounding	Does the explanation cite the required evidence?	coverage of required evidence ids (citation coverage, not faithfulness)	An explanation that never mentions the correction that resolved the conflict.

Diagnostics such as parse latency and token counts are reported separately and never enter a layer headline.

Metric definitions

Normative posterior. Each scenario has a binary physical state space $\lbrace \text{anomalous}, \text{normal} \rbrace$ with a uniform prior. An observation with confidence $c$ from a source with reliability $r$ has effective accuracy

a = 0.5 + (c - 0.5)\, r ,

and contributes $\pm \log \frac{a}{1-a}$ to the log-odds of the state it reports, provided it is applicable: same epoch as the current state, not stale, not superseded, and, for a sensor reading, routed to the target through a measures relation. Posteriors tied within $10^{-12}$ are labelled undetermined.

Trajectory score (primary). For scenario $s$ and prefix $k$ , let $\hat p_{s,k}$ be the predicted distribution and $p^{*}_{s,k}$ the reference. With total variation $\mathrm{TV}(\hat p, p^{*}) = \tfrac12 \sum_{x} |\hat p(x) - p^{*}(x)|$ (for two states, simply $|\hat p(\text{anomalous}) - p^{*}(\text{anomalous})|$ ),

\text{trajectory score} = 1 - \frac{1}{N} \sum_{(s,k)} \mathrm{TV}\big(\hat p_{s,k},\, p^{*}_{s,k}\big),

over all $N = 1{,}150$ scored scenario–prefix pairs, with a missing or malformed prediction assigned $\mathrm{TV} = 1$ . A conditional variant that averages only over produced predictions is reported as an exploratory diagnostic: it upper-bounds update quality because it conditions on a non-random subset of prefixes.

Strict state accuracy. A prefix is correct only when the predicted distribution has the same unique top state as the reference, or is tied when the reference is undetermined.

Preservation and revision. On prefixes whose reference direction is preserved, preservation accuracy is the fraction on which the predicted top state did not change. On prefixes whose reference direction is reversed, the correct-revision rate is the fraction on which the predicted top state flipped to the reference state.

Brier distance. $\tfrac12 \sum_x (\hat p(x) - p^{*}(x))^2$ , mean over present predictions (and a “full” variant with missing outputs at distance 1). It measures agreement with the normative reference distribution, not empirical calibration.

Set metrics. Attribution, localization and explanation coverage use precision, recall and $F_1$ over evidence ids. An empty gold set with an explicit empty prediction is a true negative and is excluded from $F_1$ rather than scored as $F_1 = 1$ ; false positives against an empty gold set score $F_1 = 0$ .

Actions. Exact compliance requires the reference action type; family agreement accepts any action in the same family; the evidence-supported rate asks whether the chosen action is justified by the graph evidence irrespective of the single reference action. Unsafe (shutdown, actuate) and inadmissible (ignore) choices are counted separately.

Statistical analysis

The scenario is the independent unit. Generation repeats (three seeds) are averaged within scenario before any inference.

Confidence intervals: 95% scenario-clustered bootstrap, 10,000 resamples.
Paired continuous contrasts: two-sided sign-flip permutation tests, 100,000 resamples (exact below 21 pairs).
Paired binary outcomes: exact McNemar tests on the scenario-level fully-grounded indicator.
Multiplicity: Holm correction within each preregistered family (the eleven representation contrasts, including three regime-scoped versions, and the two model contrasts, per metric).
Preregistration: all contrasts, metrics, exclusion rules, seeds, decoding settings and the runtime fallback were preregistered before the full run (preregistration.md). The conditional trajectory score and the Gemma non-canonical systems are the two labelled exceptions.

Regime-level analyses and evidence-prefix curves never treat prefixes as independent scenarios.

Run the benchmark

Everything runs offline, without robot hardware. The base package has no model dependency; torch and transformers are optional extras used only by the runtime.

git clone https://github.com/RanulfoBezerra/Evidence-Graph-Toolkit
cd Evidence-Graph-Toolkit
git checkout evaluation-v1.0
python -m pip install -e .

# Regenerate the frozen suite and run the independent log-odds checker
python experiments/prepare_evaluation_v1.py \
  --scenario-seed 20260725 --scenarios-per-regime 50 \
  --output-dir results/evaluation_v1_preparation

# Six transparent fusion baselines on the same suite
python experiments/run_belief_trajectory_benchmark.py \
  --scenarios-per-regime 50 --scenario-seed 20260725

git clone https://github.com/RanulfoBezerra/Evidence-Graph-Toolkit
cd Evidence-Graph-Toolkit
git checkout evaluation-v1.0
python -m pip install -e ".[models]"

# Seven-scenario pilot: checks runtime compatibility and captures model revisions
python experiments/run_model_matrix.py --mode pilot \
  --scenarios-per-regime 50 --max-new-tokens 512 \
  --output-dir results/evaluation_v1_pilot_512

# Frozen full run: 12 systems x 350 scenarios x 3 generation seeds (42, 43, 44).
# --mode full requires the pinned model/tokenizer revisions captured by the pilot.
python experiments/run_model_matrix.py --mode full \
  --scenarios-per-regime 50 --max-new-tokens 512 \
  --revision-manifest results/evaluation_v1_pilot_512/resolved_revisions.json \
  --output-dir results/evaluation_v1_full_512

# Preregistered statistics: bootstrap CIs, permutation tests, McNemar, Holm
python experiments/run_evaluation_statistics.py \
  --results-dir results/evaluation_v1_full_512 \
  --output-dir results/evaluation_v1_statistics

The decoding settings are fixed by the manifest: sampling at temperature 0.2, top-p 0.9, a 512-token output budget, three generation seeds treated as repeats nested within scenario. Your own model plugs in either as a HuggingFace text-generation pipeline or through a stdin/stdout command adapter; see the toolkit README for --hf-model and --command-model.

Prediction format

The runner writes one JSONL row per scenario–prefix. If you bring your own runtime, produce the same shape and score it with the frozen metrics:

{"sample_id": "evaluation-v1-contradiction_resolution-20260725-000--belief-step-3",
 "task": "belief_update",
 "output": "<raw model text>"}

The output string must contain (or repair to) the belief-update contract. For the target entity and attribute, give a normalized distribution over the two physical states, the categorical result, its confidence, and the supporting and contradicting evidence ids; list active contradictions explicitly, with an empty list when there are none:

{
  "beliefs": {
    "gauge-1": {
      "condition": {
        "distribution": {"anomalous": 0.448, "normal": 0.552},
        "most_likely": "normal",
        "state_status": "unique",
        "confidence": 0.552,
        "supporting_evidence": ["obs-1"],
        "contradicting_evidence": ["obs-2"]
      }
    }
  },
  "contradictions": [
    {
      "contradiction_type": "same_epoch_value_conflict",
      "affected_entity": "gauge-1",
      "affected_attribute": "condition",
      "conflicting_evidence_ids": ["obs-1", "obs-2"]
    }
  ]
}

When the two states are tied, set most_likely to undetermined and state_status to tied; never add undetermined to the distribution itself. Offline repair may close an unclosed code fence or truncated braces, but never fabricates field values; a repaired output counts toward the repair rate.