Dataset

The CREG evaluation-v1.0 suite is a synthetic, frozen benchmark of 350 scenarios for shared belief updating in connected robotics. Each scenario is a small evidence graph: three heterogeneous agents (a robot, an IoT sensor, an edge-AI module) report on five physical objects, and the task is to track the belief about one target gauge as the reports arrive. Because every scenario is generated from a checked-in normative specification, the correct posterior at every evidence prefix is known exactly and is shipped with the data.

350
scenarios
7 regimes × 50
1,150
scored prefixes
1,500 reference steps incl. priors
1,150
observations
3–4 per scenario
400
semantic relations
adjacent_to, measures

There are no videos, images or free text to annotate, and no train/validation/test split: the suite is an evaluation set. Labels come from the normative model, not from crowd workers; a blinded expert audit checked their plausibility.

Anatomy of one scenario

Every record in the JSONL file has the same shape. Below is a trimmed real record, the contradiction-resolution scenario used as the paper’s worked example (open it in the explorer).

Timeline of four evidence arrivals and the normative reference posterior for a contradiction-resolution scenario
Top: four arrivals with source, claim and confidence; the robot’s report contradicts the IoT sensor’s, and a delayed high-confidence correction supersedes it. Bottom: the normative reference posterior that models are scored against at every prefix, with the contradiction-active interval shaded. 
{
  "id": "evaluation-v1-contradiction_resolution-20260725-000",
  "agents": [
    {
      "id": "robot-1",
      "kind": "robot",
      "metadata": {
        "reliability": 0.78
      }
    },
    {
      "id": "iot-sensor-1",
      "kind": "iot_sensor",
      "metadata": {
        "reliability": 0.91
      }
    },
    {
      "id": "edge-ai-1",
      "kind": "edge_ai",
      "metadata": {
        "reliability": 0.52
      }
    }
  ],
  "objects": [
    {
      "id": "gauge-1",
      "label": "Pressure Gauge 1",
      "category": "gauge"
    },
    {
      "id": "gauge-2",
      "label": "Pressure Gauge 2",
      "category": "gauge"
    },
    {
      "id": "sensor-a",
      "label": "Remote Sensor A",
      "category": "sensor"
    },
    {
      "id": "sensor-b",
      "label": "Remote Sensor B",
      "category": "sensor"
    },
    {
      "id": "distractor-000",
      "label": "Distractor 000",
      "category": "panel"
    }
  ],
  "relations": [
    {
      "source_id": "gauge-1",
      "target_id": "gauge-2",
      "predicate": "adjacent_to",
      "confidence": 0.8,
      "evidence_ids": []
    }
  ],
  "observations": [
    {
      "id": "obs-1",
      "agent_id": "iot-sensor-1",
      "subject_id": "gauge-1",
      "predicate": "condition",
      "value": "normal",
      "confidence": 0.83,
      "timestamp": "2026-01-01T12:00:10Z",
      "metadata": {
        "arrival_time": "2026-01-01T12:00:10+00:00",
        "state_epoch": 1,
        "supersedes_evidence_ids": []
      }
    },
    {
      "id": "obs-2",
      "agent_id": "robot-1",
      "subject_id": "gauge-1",
      "predicate": "condition",
      "value": "anomalous",
      "confidence": 0.84,
      "timestamp": "2026-01-01T12:00:20Z",
      "metadata": {
        "arrival_time": "2026-01-01T12:00:20+00:00",
        "state_epoch": 1,
        "supersedes_evidence_ids": []
      }
    },
    {
      "id": "obs-3",
      "agent_id": "iot-sensor-1",
      "subject_id": "gauge-1",
      "predicate": "condition",
      "value": "normal",
      "confidence": 0.96,
      "timestamp": "2026-01-01T12:00:30Z",
      "metadata": {
        "arrival_time": "2026-01-01T12:00:34+00:00",
        "state_epoch": 1,
        "supersedes_evidence_ids": [
          "obs-2"
        ]
      }
    },
    {
      "id": "obs-distractor",
      "agent_id": "edge-ai-1",
      "subject_id": "distractor-000",
      "predicate": "indicator",
      "value": "nominal",
      "confidence": 0.6,
      "timestamp": "2026-01-01T12:00:12Z",
      "metadata": {
        "arrival_time": "2026-01-01T12:00:12+00:00",
        "state_epoch": 1,
        "supersedes_evidence_ids": []
      }
    }
  ],
  "evaluation": {
    "belief_trajectory": [
      "... steps 0-2 ...",
      {
        "step_index": 3,
        "new_evidence_ids": [
          "obs-2"
        ],
        "active_evidence_ids": [
          "obs-1",
          "obs-2"
        ],
        "event_types": [
          "contradiction_onset",
          "preserve"
        ],
        "reference_beliefs": [
          {
            "entity_id": "gauge-1",
            "attribute": "condition",
            "reference_state": "normal",
            "probability_direction": "weakened",
            "supporting_evidence": [
              "obs-1"
            ],
            "contradicting_evidence": [
              "obs-2"
            ],
            "reference_distribution": {
              "anomalous": 0.4485,
              "normal": 0.5515
            }
          }
        ],
        "contradictions": [
          {
            "contradiction_present": true,
            "contradiction_type": "same_epoch_value_conflict",
            "involved_evidence_ids": [
              "obs-1",
              "obs-2"
            ],
            "resolution_status": "active"
          }
        ],
        "action": {
          "reference_action": {
            "action_type": "verify",
            "target_entity_id": "gauge-1"
          }
        },
        "expected_update": false
      },
      {
        "step_index": 4,
        "new_evidence_ids": [
          "obs-3"
        ],
        "active_evidence_ids": [
          "obs-1",
          "obs-3"
        ],
        "superseded_evidence_ids": [
          "obs-2"
        ],
        "event_types": [
          "delayed_evidence",
          "contradiction_resolution",
          "preserve"
        ],
        "reference_beliefs": [
          {
            "reference_state": "normal",
            "probability_direction": "strengthened",
            "supporting_evidence": [
              "obs-1",
              "obs-3"
            ],
            "contradicting_evidence": [],
            "reference_distribution": {
              "anomalous": 0.0216,
              "normal": 0.9784
            }
          }
        ],
        "contradictions": [
          {
            "contradiction_present": false,
            "resolution_status": "resolved",
            "superseded_evidence_ids": [
              "obs-2"
            ]
          }
        ],
        "action": {
          "reference_action": {
            "action_type": "monitor"
          }
        }
      }
    ],
    "action": {
      "reference_action": {
        "action_type": "monitor",
        "target_entity_id": "gauge-1"
      },
      "admissible_actions": [
        "monitor"
      ],
      "inadmissible_actions": [
        "ignore"
      ],
      "unsafe_actions": [
        "shutdown",
        "actuate"
      ]
    }
  },
  "metadata": {
    "trajectory_regime": "contradiction_resolution",
    "suite_seed": 20260725,
    "counterfactual_pair_id": null,
    "reference_model": "controlled-binary-bayes-v1"
  }
}
FieldMeaning
agents[]Sources. kind is robot, iot_sensor or edge_ai; metadata.reliability is the source’s track record in [0.5, 1], which the normative likelihood multiplies into every report’s confidence.
objects[]Physical entities: two gauges, two remote sensors and one distractor panel. gauge-1 is always the scored target.
relations[]Semantic relations between objects. adjacent_to is context; measures (sensor → gauge, confidence 1.0) is the only route by which a sensor reading can inform the gauge’s condition.
observations[]Evidence records. Each names its source, subject, predicate and value, carries a confidence in [0.5, 1], an event timestamp, and metadata with arrival_time (may be later than the event), state_epoch, and supersedes_evidence_ids for corrections.
evaluation.belief_trajectory[]One entry per evidence prefix, step 0 being the prior. It lists which evidence is new, active, stale or superseded, the event types, the reference posterior and categorical state, the probability direction, supporting and contradicting evidence, the contradiction record and the reference action at that step.
evaluation.actionThe endpoint action labels: the reference action, the admissible set, inadmissible (ignore) and unsafe (shutdown, actuate) actions, and the rules that produced them.
evaluation.contradictions, evaluation.beliefs, evaluation.evidenceEndpoint summaries of the final contradiction state, the final belief and per-evidence provenance, times and reliabilities.
metadataRegime, case index, suite seed, counterfactual pair id (relation-association only), the reference model name and the SHA-256 of the normative specification.

What the model sees and must return. At prefix kk the model receives the scenario serialized in one representation condition, restricted to the evidence that has arrived, and the question what is the shared belief after the evidence available at this arrival step? It must return a normalized distribution over anomalous and normal for gauge-1.condition, the top state (or undetermined on a tie), supporting and contradicting evidence ids, and the list of active contradictions. Separate endpoint tasks ask for contradiction detection, the next action from verify | inspect | request_observation | monitor, and an evidence-grounded explanation. The exact contract is on the Evaluation page.

The seven regimes

Each regime is defined by a construction constraint that the generator enforces and the independent checker verifies. Fifty scenarios per regime; the third prefix is typically where the constraint bites.

Regime Construction constraint What it tests Steps Scenarios
Justified revision applicable evidence reverses the top belief Does the model reverse its top belief when strong, applicable evidence arrives? 4 50 Explore →
Belief preservation weak counterevidence does not reverse the top belief Does it hold its belief against weak counterevidence instead of flip-flopping? 4 50 Explore →
Relation association measures relation is necessary to associate sensor evidence with the target Can it route a sensor reading to the gauge only through the declared measures relation? Counterfactual twin: same observations, swapped relations. 4 50 Explore →
Contradiction onset opposing claims in one epoch create an active contradiction Does it detect and localize two opposing same-epoch claims? 4 50 Explore →
Contradiction resolution correction evidence supersedes a conflicting claim Does it drop a superseded claim when a correction arrives, and mark the contradiction resolved? 5 50 Explore →
Stale / delayed evidence older-epoch late evidence is stale while delayed current-epoch evidence applies Does it ignore an older-epoch report that arrives late, yet apply a delayed current-epoch report? 5 50 Explore →
Source reliability source reliability changes the posterior relative to confidence-only fusion Does it weight a report by who sent it, not just by its stated confidence? 4 50 Explore →
Total (suite seed 20260725; 3 generation seeds; 512-token budget) 4–5 350

Steps include the prior (step 0). Every scenario also carries one distractor observation about an unrelated panel that must be isolated from the target belief.

Counterfactual pairs. The 50 relation-association scenarios form 25 pairs. Within a pair the observations are byte-identical; only the measures relations are swapped, so in one twin the sensor reading routes to gauge-1 and the reference belief moves, while in the other it routes to gauge-2 and the reference belief must stay undetermined. Delete the relations and the twins become indistinguishable, so any score difference on them is attributable to relation content alone. Compare case 000 with case 001.

The normative model

Every label is derived from a short specification (normative_spec.json, download) rather than annotated by hand.

  • State space and prior. Two physical states, anomalous and normal, with a uniform prior. Ties within 10−1210^{-12} are reported categorically as undetermined while the distribution stays over the two physical states.
  • Likelihood. A report with confidence cc from a source with reliability rr has effective accuracy a=0.5+(c−0.5) ra = 0.5 + (c - 0.5)\,r and shifts the log-odds by ±log⁡a1−a\pm\log\frac{a}{1-a} toward the state it reports. Confidence and reliability both live in [0.5,1][0.5, 1].
  • Epochs and time. Reports carry a state_epoch. Opposing claims in the same epoch contradict. A report from an older epoch that arrives late is stale and is ignored; a delayed report from the current epoch still applies.
  • Supersession. A correction names the evidence it supersedes; the superseded claim leaves the active set, which also resolves any contradiction it was part of.
  • Relation transfer. A sensor reading informs a gauge’s condition only through a measures relation from that sensor to that gauge, with relation confidence 1.0.
  • Contradictions. Type same_epoch_value_conflict: onset when two active, applicable claims disagree; resolution when no active conflict remains after supersession or an epoch change.
  • Reference actions. verify while a contradiction is active; otherwise inspect if the top state is anomalous, monitor if normal, request_observation if undetermined. shutdown and actuate are unsafe; ignore is inadmissible.

An independent checker reconstructs every prefix posterior in log-odds space without importing the generator’s updater. The frozen suite passes on all 1,500 reference steps.

Representation conditions

Every system receives the same facts; conditions differ only in how evidence is serialized. Four principal conditions are crossed with all models, and five factor ablations run on Qwen2.5-7B only.

ConditionRoleDescriptionMean prompt length
| **Canonical JSON** | principal | Relation triples in flat JSON. Same facts, shortest prompts (0.843× the full-graph length). | 5,284 chars (0.843×) | | **Structured list** | principal | Records as an information-matched structured list. | 5,651 chars (0.901×) | | **Graph without relations** | principal | Graph serialization with semantic-relation content deleted. | 6,058 chars (0.966×) | | **Full graph** | principal | Agents, evidence, entities and semantic relations serialized as a graph. | 6,271 chars (1.000×) | | **Flat, edge-removed relations** | ablation | Relation content retained, graph serialization removed (length-matched control, 1.004×). | 6,298 chars (1.004×) | | **No provenance** | ablation | Removes reported edges and the source-reliability field (a joint manipulation). | 5,834 chars (0.930×) | | **No timestamps** | ablation | Removes timestamps but keeps explicit state epochs (intentionally weak). | 6,053 chars (0.965×) | | **No confidence** | ablation | Removes per-report confidence. | 6,210 chars (0.990×) | | **Shuffled full graph** | ablation | Identical records in permuted order, fixed shuffle seed (exact control, 1.000×). | 6,271 chars (1.000×) |

Two conditions are exact controls: the shuffled graph is character-identical to the full graph up to ordering, and flat edge-removed nearly so. Two manipulations carry disclosed confounds: canonical JSON encodes the same information in shorter prompts, so serialization and length covary; and no-provenance also removes the reported edges and the source-reliability field the normative likelihood consumes. No-timestamps keeps state epochs, so temporal semantics stay recoverable and the manipulation is intentionally weak. There is no dedicated source-reliability ablation.

Below is the worked-example scenario rendered verbatim by the toolkit’s frozen renderers in the four principal conditions (full prompt bodies; the task instructions and question are added around them).

The same observation record excerpted from the prompts of the four principal representation conditions
One observation from the worked example, excerpted verbatim from the frozen prompts of the four principal conditions. All carry the same facts; only serialization differs, except graph-without-relations, which deletes semantic-relation content.

How the suite was built and frozen

  1. Specify. The normative specification fixes the state space, prior, likelihood, temporal and supersession rules, relation transfer, contradiction rules, action rules and one construction constraint per regime.
  2. Generate. The generator (evaluation-v1.0-generator.2) draws 50 scenarios per regime from suite seed 20260725, rejecting duplicate semantic fingerprints, and derives every reference label from the specification.
  3. Check independently. A separate log-odds checker (evaluation-v1.0-log-odds-checker.2) recomputes every prefix posterior without the generator’s code. Zero mismatches.
  4. Audit. Before any full model run, one independent expert reviewed a stratified sample of 35 scenario–prefix pairs (five per regime) blinded-then-reveal: they recorded the expected top belief, direction, supporting and counter-evidence and contradiction status first, locked the answers, then compared against the oracle. Round one accepted 29 and required corrections on 6, concentrated in the relation-association, source-reliability and stale/delayed regimes. The cause (tied-posterior semantics) was repaired in the specification, the suite regenerated, and round two accepted all 35. Labels were never patched by hand. The audit is a semantic-plausibility sanity check; it does not establish human consensus.
  5. Pilot. A seven-scenario runtime pilot selected the 512-token output budget from the preregistered candidates and captured immutable model and tokenizer revisions.
  6. Freeze. Seven gates had to pass before the suite, references, prompts, renderers, metrics and runtime configuration were tagged evaluation-v1.0. Post-freeze corrections require a changelog entry and protocol increment and never silently change a score.
  • Unique scenarios
    350 unique scenario IDs and semantic fingerprints
  • Independent reference checker
    independent checker agrees on every prefix
  • Relation counterfactual pairs
    all relation pairs are observationally indistinguishable without relations
  • Expert semantic-plausibility audit
    35 scenario-prefix expert sanity checks (five per regime), complete and without unresolved corrections
  • Runtime pilot
    seven-scenario model/representation pilot meets all thresholds
  • Repository tests
    full repository tests pass
  • Output contract tests
    all task output contracts pass

Statistics

Scenarios per regime
  • Justified revision 50
  • Belief preservation 50
  • Relation association 50
  • Contradiction onset 50
  • Contradiction resolution 50
  • Stale / delayed evidence 50
  • Source reliability 50
Final reference action
  • verify 200
  • inspect 75
  • monitor 75
Final probability direction
  • reversed 150
  • weakened 75
  • preserved 75
  • strengthened 50
Event types across all steps
  • update 525
  • isolate 400
  • initial 350
  • delayed evidence 325
  • contradiction onset 250
  • preserve 225
  • relation association 50
  • contradiction resolution 50
  • stale evidence 50
Observation values
  • normal 400
  • anomalous 400
  • nominal 176
  • unknown 174
Relation predicates
  • adjacent_to 300
  • measures 100
Scenarios350 (7 regimes × 50)
Scored scenario–prefix pairs1,150 (1,500 reference steps including 350 priors; 250 scenarios with 4 steps, 100 with 5)
Agents3 per scenario, one of each kind (robot, iot_sensor, edge_ai)
Objects5 per scenario: 2 gauges, 2 remote sensors, 1 distractor panel
Observations1,150 (250 scenarios with 3, 100 with 4); predicates condition, indicator, reading
Relations400 (300 adjacent_to, 100 measures)
Counterfactual pairs25 (relation-association regime)
Final reference state200 normal / 150 anomalous
Seedssuite seed 20260725; generation seeds 42, 43, 44 (decoding repeats, nested within scenario)
Decoding (frozen)sampling, temperature 0.2, top-p 0.9, 512-token output budget
Annotation sourcenormative model (derived); no human-written labels

How CREG differs from prior benchmarks

Belief-R BeliefTrack † BayesBench † ALFRED / TEACh / B-1K OpenEQA CREG
Evidence setting
Sequential evidence with revision pressure present present present absent absent present
Reports attributed to distinct physical sources with reliability absent absent absent absent absent present
Per-report confidence, provenance, and timestamps absent absent absent absent absent present
Stale, delayed, and superseded evidence semantics absent absent absent absent absent present
Labeled contradictions (detection, localization, type) absent absent absent absent absent present
Relation-dependent cases with counterfactual pairs absent absent absent absent absent present
Evaluation targets
Probabilistic reference posterior at every evidence step absent partial present absent absent present
Belief preservation / unnecessary updates penalized present present partial absent absent present
Entity grounding scored separately from belief quality absent absent absent absent absent present
Action admissibility separated from exact policy compliance absent absent absent partial absent present
Evidence-grounded explanations scored absent absent absent absent absent present
Information-controlled representation ablations absent absent absent absent absent present

present · partial · absent. Cell values follow the published descriptions of each benchmark. † marks concurrent preprints.

Belief-revision benchmarks such as Belief-R, BeliefTrack and BayesBench test sequential updating in text, but without physically attributed sources, reliabilities, timestamps, staleness or labelled contradictions. Embodied benchmarks such as ALFRED, TEACh, BEHAVIOR-1K and OpenEQA test acting and answering in rich environments, but do not expose a probabilistic reference belief at every evidence step or score grounding, attribution and explanation separately. CREG is deliberately narrow: exact references and counterfactual control in exchange for ecological richness.

Limitations

  • Synthetic and binary. Scenarios come from a normative specification with a two-state space and a single scored target entity. This buys exact references and counterfactual control at the cost of ecological richness. The toolkit’s physical-log replay demonstrates the ingestion pathway on recorded observations but is a workflow demonstration, not an empirical robot evaluation.
  • One expert. The plausibility audit used one independent expert under a blinded-then-reveal protocol. It screens for generator and specification errors; it does not establish human consensus or model–human agreement, and no inter-rater statistics exist.
  • Confounded ablations. No-provenance jointly removes reliability; canonical JSON is also shorter; no-timestamps is intentionally weak; there is no dedicated source-reliability ablation.
  • Exploratory extras. The conditional trajectory score and the Gemma runs outside canonical JSON were not preregistered and the paper’s conclusions do not rest on them.
  • Panel size. Four open-weight instruction models between 1.5B and 12B parameters at a fixed 512-token budget. Larger models, tool-augmented systems and longer budgets are untested.

License and maintenance

The suite, references, prompts and code are released under the MIT License. The data is synthetic and contains no personal information. The benchmark is frozen at evaluation-v1.0; issues and submissions go through the toolkit repository. See the Download page for files, hashes and citation.