Abstract
Connected physical systems fuse observations from robots, IoT sensors, and edge-AI modules into a shared account of the world, yet existing evaluations rarely distinguish whether a reasoning method attributed the evidence, updated its beliefs, handled contradictions, and justified its action, or merely produced an acceptable final answer. We introduce the Connected Robotics Evidence Graph (CREG) Benchmark, a frozen, preregistered benchmark and layered evaluation protocol for evidence-grounded shared belief updating. The suite comprises 350 deterministically generated scenarios across seven regimes, each exposing per-prefix normative posterior references validated by an independent log-odds checker and screened by a blinded expert semantic-plausibility audit. Nine information-controlled representation conditions separate relation content from serialization, ordering, and metadata fields. Evaluating four open-weight instruction models against six transparent fusion baselines, we find that entity grounding separates the panel before belief updating does; that relation content matters exactly where it is constructed to matter, while graph topology per se confers no measured advantage; and that belief accuracy is strongly non-monotonic along the evidence sequence, exhibiting mid-trajectory failures that endpoint-only evaluation cannot detect.
What CREG asks
A pressure gauge is either anomalous or normal. A mobile robot, an IoT sensor and an edge-AI module each report on it, or on something nearby, with their own confidence and their own track record. Reports arrive out of order; some are stale, some correct earlier ones, some contradict each other, and some concern a sensor that only matters because a measures relation ties it to the gauge.
After each arrival, the system must state the shared belief over the gauge’s condition as a probability distribution, say which evidence supports and contradicts it, flag any active contradiction, and choose an admissible next action. CREG generates every scenario from a checked-in normative specification, so the correct posterior at every step is known exactly, and the benchmark scores the whole trajectory rather than the final answer.
Three-minute explainer
What we found
Weaker models lose most of their trajectory score to outputs that never name the required entity. Conditional on grounding, their updates approach the stronger models’.
Deleting relations costs 15.3 points of strict accuracy on relation-dependent scenarios, yet a structured list beats the full graph and shuffling the graph does not hurt.
Accuracy collapses to 0.20–0.25 at the third prefix and recovers by the fourth. Endpoint-only evaluation cannot see it.
Start here
Watch a contradiction open at step 3 and resolve at step 4 as a delayed correction supersedes the robot’s report. Every one of the 350 scenarios is browsable.
Record schema, the seven regimes, the normative model, the nine representation conditions, how the suite was built and audited, and how it compares to prior benchmarks.
Twelve systems, six fusion baselines, preregistered paired contrasts with Holm-corrected p-values, and how to submit your own run.
The frozen JSONL suite with SHA-256, the manifest, the normative specification, a one-command regeneration recipe and BibTeX.
Everything reproduces offline, without robot hardware: scenario generation, normative references, the independent checker, prompts, per-scenario raw outputs, the statistical analysis and every figure and table. The benchmark is frozen under the immutable git tag evaluation-v1.0; post-freeze corrections are documented in the changelog and never change scenarios, references or prompts.