X-Ray: Numerical Evidence and Divergence Attribution

X-Ray is CKE's evidence system for numerical correctness. It compares the generated CK runtime against pinned oracle backends checkpoint by checkpoint, joins the results with the call IR and the kernel registry, and reduces every failure to one first divergent semantic edge, one failure class, one named fix owner, and one recommended action.

The Evidence Pipeline: Join, Don't Instrument

The design principle is join, don't instrument. The call IR already carries every operation with its resolved kernel and contract. X-Ray reports already carry per-stop metrics. The kernel registry is already loaded. X-Ray connects the three instead of adding a parallel instrumentation stack inside the runtime.

The binding begins before any tensor exists. A circuit declares versioned semantic checkpoints; BuildIR binds each checkpoint to a generated operation, an exact kernel ID, a public function, a phase, a layer, a layout and named axes. GraphIR, LoweredIR and call IR retain the same metadata, so the checkpoint-manifest adapter takes tensor identity from the call IR rather than from backend-specific file names. Backend profiles hold observed storage policy and dtype tolerances outside the circuits.

The ordering rule is strict: metadata contracts are validated before tensor values are loaded. A storage, layout, reduction, position or execution-policy mismatch stops the comparison before any expensive model execution or numerical metric runs — a dtype disagreement is a contract fault, not a large RMSE. Named axes canonicalize compatible physical layouts, sparse checkpoints identify a failing interval, and the planner then requests only the internal edges of the first failing block.

What a Checkpoint Record Carries

Every captured artifact is self-describing and content-addressed. A trimmed real record from the checked-in fixture (version/v8/tests/fixtures/xray/xray_execution_trace.json, schema cke.xray_execution_trace):

{
 "checkpoint_id": "layer.0.attention.kv_append",
 "role": "new_key",
 "tensor_path": "tensors/layer0_new_key.f32",
 "dtype": "fp32",
 "shape": [1, 8, 1, 64],
 "sha256": "1a2b3c4d…"
}

state: position 8 · cache_token_count 9 · append_index 8
cache_layout: kv_cache_v1 · layer/head/token/channel strides

The execution trace records the schedule alongside the tensors: prefill segmentation, per-call kernel batches with kernel ID and numerical/effective contract IDs, cache actions, position progression, and the physical cache layout with its strides. Two runs whose schedules differ are different executions, and X-Ray says so before comparing a single value — a combined 1,307-row prefill is not numerical evidence about a backend that executes 33, 1,008 and 266-row segments.

Verdict Semantics and the Honest Scorecard

Every report lands in one of three states:

When a backend reaches full parity the board may say so, but the claim stays scoped. The honest banner is 79/79 edges within gate — FP32 · PyTorch · fixture 20260725 · 1 thread, never a bare "100%". Parity is always scoped to model family, dtype, backend version, fixture or seed, thread count and sample. Live mode keeps the scorecard a continuously re-proven state: the next agent run that regresses an edge flips the card back to amber or red with the first-divergence pin.

Anatomy of the Drift Chart

The visualizer's drift chart is the fastest way to read a run: stop index in execution order on the x-axis, log-scale max_abs and RMSE on the y-axis, one curve per loaded report, layer facets behind the points. The markers are the language of the chart:

• Green byte-exact prefix — checkpoints that match the oracle exactly; the prefix length is itself evidence.
• Amber pin — first non-exact checkpoint: where exactness ends, not necessarily where the bug is.
• Purple ▲ — a per-edge error jump greater than 3×: the edge where drift accelerates.
• Magenta ◆ — a monotonicity break in the error curve.
• Red pin — first material divergence: a top-1 logit flip, the point where generation behavior changes.
• Gray circuit rows — ops without a checkpoint stay visible; the header states coverage explicitly (N/M ops have X-ray edges) so a gap can never masquerade as a pass.

Capture Neutrality: Evidence Must Not Change the Answer

Checkpoint export is additional work inside the observed process — environment checks, memory copies, path formatting, file I/O between model operations. That work changes timing and cache state, and an existing race, unsafe scratch-buffer lifetime or scheduling-dependent reduction can become visible only when instrumentation perturbs the run. A plausible tensor dump is not sufficient: a captured tensor is evidence only if the captured execution reproduces an uncaptured execution on the same causal token history.

The neutrality comparison is bit-exact on purpose, and only within the same runtime. Two executions of the same CKE runtime, provider map, thread count and forced history have a strict contract: instrumentation must not change any output bit. A numerical tolerance here could hide precisely the small perturbation a deep recurrent model later amplifies. Cross-engine comparison is a different question — CKE and llama.cpp may legitimately use different reduction orders — so oracle parity uses numerical metrics and token decisions, never a bit-exact demand. If a production provider is intentionally nondeterministic, that fact must first be measured and declared as its own contract; X-Ray must not silently absorb it into a tolerance.

Failure Classes

Classification is a closed vocabulary — every failure resolves to one class with a recommended action, and metadata faults never proceed to numerical metrics:

The nightly injected-fault validator tests this vocabulary directly: public two-layer fixtures each change one tensor, metadata field, circuit edge or state transition, and X-Ray must report the predetermined first boundary — a BF16 storage boundary must come back as STORAGE_CONTRACT_MISMATCH, an off-by-one KV append index as CACHE_STATE_METADATA_MISMATCH, and so on. A clean control run must produce no false positive. The diagnostic itself is under test, not only the models.

Fix Ownership: Where a Diagnosis Is Allowed to Land

X-Ray is an attribution tool, not permission to hardcode the observed answer into code generation. Every accepted parity fix has exactly one explicit owner:

The DSL and code generator remain a deterministic consumer of circuit requirements and resolved kernel-map decisions. No model-name, checkpoint-name or one-off parity branches in DSL/codegen: if X-Ray reports a model-specific mismatch, the fix lands in the circuit requirement, the kernel capability/map, the kernel arithmetic, or the reference adapter. Reports encode this rule in first_divergence.fix_owner and architecture_policy, so the policy arrives with the failure instead of living only in documentation. The full additive fix progression — execution policy first, cache state before attention math, first divergent edge, rerun from the last passing checkpoint — lives in v8 Numerical Contracts.

Report Taxonomy

Each X-Ray artifact is schema-validated JSON with a declared schema ID and version; unknown or malformed evidence fails closed. The principal report types:

The Visualizer Surface

The IR visualizer's X-Ray tab is where a human watching an agent bring up a model sees, in seconds, which backend diverges, at which op, under which kernel contract, and whether it flips output tokens:

• Backend status board — one card per loaded report: backend chip, three-state verdict, first non-exact and first material divergence stops, thresholds, provenance, source path.
• Run context bar — backend, phase badge (PREFILL / DECODE / TEACHER_FORCED), seed, threads, backend version; a phase toggle appears when two or more phases are present.
• Drift chart — the marker language above, with clickable checkpoints and an export-SVG button for docs-ready figures.
• Circuit X-Ray — one row per call-IR op in execution order, joined with checkpoints at the same stop: provider chip (click jumps to Kernel Flow), required-vs-resolved contract chip (amber = substituted), log-severity heat cells for max_abs, RMSE, exact_ratio, phase.
• Ranking strips — per-position top-1 agreement grouped by check kind.
• Execution trace and state stages — ordered divergence stages with the first broken stage and what it poisoned downstream.
• Operator runbook — pre-filled reproduction commands for the report's run directory.
• Live mode — X-Ray filenames are registered in live polling, so the tab hot-reloads while agents emit new reports.

Known gaps are disclosed rather than smoothed over: the ranking-to-drift overlay is intentionally not rendered (token positions do not map cleanly onto op stops); the phase toggle has not been visually verified against real multi-phase data; rendered evidence so far comes from checked-in fixtures; failure-narrative sequencing and bug-bundle export are designed but not built.

Gates and Commands

# lightweight gate: schema, resolver, canonicalization, classification, bisection, manifest tests
make test-bf16-xray

# validator self-test: injected faults must produce the predetermined first diagnosis
make test-xray-validator-selftest

# real diagnosis, one pinned oracle per invocation
make xray-vision-parity BACKEND=llamacpp GGUF=<mmproj.gguf> XRAY_OUTPUT_DIR=build/xray/qwen3vl_llamacpp
make xray-vision-parity BACKEND=pytorch CHECKPOINT=<hf_ckpt> RUNTIME_DIR=<dir> \
  WEIGHTS_BUMP=<weights.bump> CALL_IR=<call.json> XRAY_OUTPUT_DIR=build/xray/qwen3vl_bf16

The lightweight gate downloads no model weights and is included in make test-bf16 and the BF16 nightly category; the injected-fault validator runs nightly as a dedicated test-report row. End-to-end: run the same model through N backend-pinned scripts, place the N schema-validated JSONs in the run directory, regenerate or live-serve the visualizer — the board groups cards by backend automatically.

X-Ray in the Wider Architecture

X-Ray is the evidence layer other CKE subsystems report into:

• Kernel Maps — provider selection is recorded by ID: the selected provider plus every rejection reason, proving compatibility filtering precedes priority ranking. Unknown or legacy identity stays null and disclosed, never inferred from filenames.
• v8 Numerical Contracts — the additive fix progression, the execution-contract X-Ray (segmentation, cache and position transitions, runtime hashes), the measured BF16 vision progression, and forward-sensitivity probing.
• Memory Tetris — the proposed storage-tier X-Ray fields (request/first-byte/completion/first-use timestamps, prefetch lead, exposed wait, residency, eviction reason, checksum and provenance status) extend the same evidence discipline to tensor transfers; proposed, not implemented.
• CKU — one public CKU with X-Ray explaining the execution composition behind it.
• Test Report — scoped scorecards and the nightly injected-fault validator row.
• Divergence Harness — the layer-by-layer comparison workflow X-Ray evidence feeds.