v8 Numerical Contracts

v8 now treats numerical behavior as compiler input. A circuit does not merely say “run attention.” It can require the query precision, K/V storage and compute precision, score accumulator, online-softmax state, value accumulator, partition threshold, partial storage, and merge order. Kernel maps state which complete contracts their implementations support. The DSL resolves one provider or fails before C is emitted.

Why the Architecture Needed Hardening

The earlier v8 path had strong kernel maps, generated C, and E2E tests, but some numerical semantics were inferred inside runtime code. A function name established one part of the contract, an environment flag selected another part, tensor metadata implied a third part, and a sequence-length threshold could silently change the reduction algorithm. Leaf kernels could all pass while the stitched model selected the wrong public route.

The Attention Bug That Exposed It

Three full-attention tests had long-standing errors of roughly 5.68e-4 to 1.06e-3. The public full-attention contract expected FP32 Q with K/V rounded through FP16, matching the GGML-style vision path. The optimized AVX2 route consumed raw FP32 K/V. Pre-rounding only K/V reduced the error to 4.77e-7, proving this was a contract mismatch rather than a tolerance problem.

Compiler Inputs

Complete Reduction Semantics

A reduction ID is not a loose dtype such as fp16 or strict. Those names are rejected as ambiguous. A registered contract includes:

• query and key compute precision;
• score accumulator and softmax-statistics precision;
• value compute and accumulator precision;
• partial storage and merge order;
• partition kind, threshold, and chunk policy.

{
  "id": "f16_online_fp32_merge",
  "q_compute": "fp16_rounded",
  "k_compute": "fp16",
  "score_accumulator": "fp32",
  "softmax_statistics": "fp32",
  "value_accumulator": "fp16_rounded_each_update",
  "partial_storage": "fp16",
  "partial_merge": "fp32_chunk_order",
  "partition": {
    "kind": "kv_chunks_by_workers",
    "threshold": 512
  }
}

Deterministic Resolution

1. The builder hydrates the current built-in circuit into cached manifests, so old converted models receive current graph and contract defaults.
2. The resolver reads each active required_contracts entry for the requested phase.
3. It matches op, phase, tensor dtype, layout, and complete reduction ID against executable kernel maps.
4. Zero providers fail as unsupported. Multiple providers fail as ambiguous.
5. The resolver selects the provider before GraphIR construction. Contract-bearing operations do not consult legacy circuit overrides or heuristic dispatch.
6. GraphIR records requirements and the resolved kernel/contract IDs. LoweredIR and call-ready IR carry that decision and hard-fail if a later stage changes it.
7. Production promotion additionally requires validated circuit, contract definition, implementation route, stitched parity, and E2E evidence.

Compatibility and Migration

The repository directory is now version/v8/circuits. Embedded BUMP metadata continues to use the historical template key, and chat templates keep their established name; changing serialized formats is not required to correct compiler architecture.

The authoritative migration preserves executable behavior while making the decision visible. Contract metadata is added to GraphIR, LoweredIR, and call-ready IR, while kernel IDs, memory layout, call arguments, and generated C remain unchanged. Gemma3, Qwen2, Qwen3, Qwen3.5, Nanbeige, and Qwen3-VL attention routes resolve without legacy selection.

Call ABI Ownership

A resolved kernel map now owns its ordered C call ABI through call_abi.version and call_abi.params. Lower 3 consumes that exact kernel-ID-owned declaration and records its owner in call-ready IR. A contract-governed operation may not fall back to kernel_bindings.json or kernel_bindings.overlay.json; missing ABI metadata, unknown source namespaces, function drift, and duplicate map/legacy ownership hard-fail.

The two legacy binding registries remain only for uncontracted compatibility helpers while migration continues. They are not authoritative inputs for attention reductions, BF16 storage boundaries, vision M-RoPE, or Q4/Q6 numerical execution contracts.

Compiler Policy Gate

make v8-regression-fast begins with an AST-based DSL policy audit. Generic compiler and code-generation entry points may branch on exact operation IDs, resolved kernel IDs, tensor metadata, and declared capability fields. They may not branch on model, architecture, or family names. Aliasing a family value through local variables does not bypass the audit.

The builder also treats operation families as exact inputs. A missing attention_sliding implementation cannot silently resolve to ordinary attention; missing and ambiguous routes fail with an instruction to fix the circuit or kernel map. Per-layer dimensions are validated arrays, circuit weight policies own aliases and intentionally unused tensors, and unknown policy fields fail schema validation.

The same policy now protects v7 training and backprop through make test-v7-dsl-policy, which is part of make v7-regression-fast. This does not make v7 and v8 share one graph, but it enforces the same ownership rule: the training IR owns backward ordering and the emitter may not acquire model-family behavior.

make test-v8-dsl
make v8-regression-fast
make test-v7-dsl-policy
make v7-regression-fast

Remaining Migration Debt

The 76-site inventory is deliberately broader than the previous 41-site estimate: it counts executable module-level and function-level string literals containing model-family names across the four principal compiler and code-generation modules, while excluding comments, docstrings, and standalone documentation strings. Not every site selects mathematics, but every site is visible debt that cannot increase silently. The audit reports a per-function inventory so cleanup can proceed in bounded groups instead of relying on an informal grep count.

The next priority is Qwen3-VL bridge generation: multimodal fallback injection, deepstack handling, M-RoPE calls, and position advancement should be explicit call-IR operations. Kimi/DeepSeek MLA decode and prefill selection follows after both phases have exact kernel-map providers. Tokenizer adapters may retain family knowledge only while hydrating external metadata into canonical configuration.

This refactor prevents four concrete failure classes: model names silently changing tensor dimensions, undeclared weights being ignored by family branches, unavailable sliding attention falling back to ordinary attention, and generated runtimes enabling multimodal or quantized behavior from a family-name test instead of a hydrated capability.

Tokenizer and format adapters may identify an external model family while translating source metadata into the canonical circuit/config representation. That knowledge must stop at hydration. It is not permission for GraphIR construction, lowering, memory planning, or codegen to select mathematics from a family name.

Threading Is an Execution Contract

Kernel maps also declare ISA dispatch, threading runtime, possible work partitions, dispatch mechanisms, and whether scheduling can alter reduction order. Q4/Q6 GEMM and GEMV maps therefore expose serial, row, column, or output-tile policies instead of hiding threadpool behavior in generated wrappers. GraphIR records this as resolved_execution, and LoweredIR plus call-ready IR hard-fail if the kernel ID changes. When partitioning changes arithmetic order, the numerical contract must define the corresponding reduction and merge behavior.

Scalar, Production, and Threadpool Evidence

Quantized linear maps bind three exact functions: the scalar contract oracle, the public production function, and the threadpool dispatch function. The compiler validates these names and carries them into IR; it does not rank or guess functions from ISA, dimensions, or benchmark results. Dimensions and thread counts remain ordinary function inputs.

1. Compare an external backend or independent mathematical oracle with the scalar contract implementation.
2. Compare the optimized public function with that scalar implementation.
3. Compare threadpool dispatch with the same scalar implementation at representative thread counts and shapes.

Independent output rows or tiles must preserve each output's reduction order. Split-K is a different numerical contract and must declare its partial accumulator, partition, and merge order. Unavailable llama.cpp or PyTorch evidence is recorded as unresolved, never counted as a pass.

X-Ray Fix Progression

Every X-ray failure follows the same additive progression. The report embeds these steps so another agent does not repeat an already diagnosed numerical bug. X-Ray: Evidence and Divergence Attribution explains the pipeline, capture neutrality and fix-ownership policy this progression runs under.

1. Compare execution policy first: prefill segmentation, kernel batch shapes, position progression, cache reset/append behavior, and cache strides. A combined 1,307-row prefill is not numerically equivalent evidence for a backend that executes 33, 1,008, and 266-row segments.
2. For incremental decode, prove cache state before attention math: compare append index, current post-RoPE K/V, stored-row readback, the previous row, and hashes for every valid cache row.
3. Only when query and valid cache bytes agree, compare attention providers, split thresholds, worker partitions, probability/value rounding, and merge order.
4. Stop at the first divergent semantic edge and classify its storage, compute, reduction, position, layout, circuit, binding, and threading contract.
5. Resolve an exact compatible kernel-map capability. Zero or multiple providers hard-fail.
6. Fix the existing implementation or add one new additive variant. Do not alter unrelated numerical variants.
7. Extend the kernel-family matrix across input storage, compute and accumulator precision, reduction and merge order, rounding points, output storage, and threading semantics.
8. Test the leaf kernel against an independent formula and the requested PyTorch or llama.cpp backend oracle.
9. Register the exact validated function in the kernel map, then run isolated, contract-resolution, stitched-checkpoint, mixed-prefill, and teacher-forced gates.
10. Rerun X-ray from the last passing checkpoint. The accepted fix must move the first divergence to a later semantic edge.
11. Publish the capability evidence in the test-report accordion and nightly artifacts before promoting the route.

The bounded execution-state comparator preserves this order in a machine-readable report:

python version/v8/scripts/xray_execution_state_v8.py \
  --subject-trace build/xray/ck-execution-trace.json \
  --oracle-trace build/xray/llama-execution-trace.json \
  --output build/xray/execution-state-report.json

It stops at one of four useful boundaries: execution-policy mismatch, cache metadata/content mismatch, attention-input mismatch, or attention-arithmetic mismatch. Tensor bisection begins only after the execution and state contracts agree.

Testing the Diagnostic, Not Only the Model

X-ray has its own nightly injected-fault gate. The fixtures are small public two-layer vision/decoder circuits; no production kernel contains a hidden fault switch. Each fixture changes one tensor, metadata field, circuit edge, or state transition and requires X-ray to report the predetermined first semantic boundary.

make test-xray-validator-selftest
# Evidence: build/xray/validator_selftest_report.json

The gate also runs a clean control. It fails if X-ray reports an earlier false positive, accepts an incompatible execution contract, misses the injected edge, or misclassifies state corruption as kernel arithmetic. The nightly test-report publishes this as a dedicated v8 X-ray Injected-Fault Validator row.

Measured BF16 Vision Progression

The Qwen3-VL BF16 investigation demonstrates the closure rule on a real 1152×896 form image with 4,032 visual tokens. Each row was reached through the normal compiler and exact kernel-map resolution; no generated C was edited by hand.

The next edge is always explicit. After output projection the resolver selects the BF16 residual capability, then X-ray continues through norm2, MLP, spatial merge, projector, final prefix, mixed-prefill logits, and teacher-forced generation. A green leaf test without this forward movement is evidence for a kernel only, not closure of the model bug.

Forward Sensitivity Before Backward Attribution

A small BF16 forward difference can be numerically harmless or it can be amplified by attention, normalization, or a later residual path. X-ray can now test that distinction without first trusting a CK backward kernel. The sensitivity runner sends both the PyTorch reference tensor and the CK-perturbed tensor through the same authoritative PyTorch operator, applies one deterministic upstream probe, and reports forward and vector-Jacobian-product amplification.

python version/v8/scripts/xray_attention_sensitivity_v8.py \
  --reference-q build/xray/torch-q.f32 \
  --reference-k build/xray/torch-k.f32 \
  --reference-v build/xray/torch-v.f32 \
  --subject-q build/xray/ck-q.f32 \
  --subject-k build/xray/ck-k.f32 \
  --subject-v build/xray/ck-v.f32 \
  --q-shape 16 4032 72 \
  --kv-shape 16 4032 72 \
  --storage-dtype bf16 \
  --query-start 0 --query-count 32 \
  --output build/xray/attention-sensitivity.json

The report performs Q-only, K-only, V-only, and combined perturbation ablations. It records the stored input delta, downstream output delta, fixed-probe gradient delta, amplification ratios, and worst logical coordinates. Query bounding limits memory while retaining the complete K/V context for full non-causal vision attention.

This report is diagnostic evidence, not a parity verdict. It proves which existing forward perturbation is amplified by PyTorch's downstream math. It does not prove that a CK backward provider is correct. A CK training provider must separately declare the same full-attention storage, accumulation, reduction, rounding, and threading contract and pass direct gradient parity. Pass/fail thresholds remain in parity profiles rather than circuits or the sensitivity report.

Execution-Contract X-Ray

Tensor parity cannot diagnose a backend that executes the same mixed prefix with a different schedule. X-ray therefore records prefill segmentation, cache transitions, position transitions, exact shared-library identity, and the manifest-map identity alongside numerical checkpoints.

On the 1,008-token Qwen3-VL form cases, the old single-call path diverged from llama.cpp as early as generated step 4. Circuit-driven segmented append removed the cache-state failure, but a later close-ranking flip remained. X-ray then showed that the visual M-RoPE cache was exact while only the rotated second half differed. Basis-vector replay isolated two compiler-visible rounding points: llama.cpp rounds the partner product before each fmaf, and its frequency recurrence uses the system powf, sinf, and cosf behavior. CK now names that evaluation order in the numerical contract and implements it explicitly.

The resulting production gate is exact for both canonical OCR forms: each full 1008 x 16384 decoder-facing visual prefix matches llama.cpp across all 16,515,072 FP32 values, and both 128-token greedy runs match 128/128 top-1 tokens. Minimum logits cosine was 0.98892 and 0.99168 respectively, with at least 14/16 top-k overlap. This closes these two fixtures, not the unexecuted 40-image sweep or every multimodal family.

The investigation also invalidated an earlier packed Q4_K×Q8_K diagnosis. The apparent packed-reduction failure was downstream of the M-RoPE ULP and a report that recorded the generated decoder library but not its dynamically loaded engine library. X-ray now defaults the raw llama.cpp oracle to one thread, rejects multi-threaded exact capture without an explicit opt-in, and records the engine, generated model, llama shim, and manifest identities. A downstream kernel is not blamed until exact-input replay and complete runtime provenance both pass.

Validation Commands

make test-numerical-contracts is an executable gate, not only a schema check. It builds the engine and runs contract resolution, full attention numerics, and FP16 split-KV reduction cases. The focused commands below remain useful when attributing a failure.

make test-numerical-contracts
make test-threadpool-parity
PYTHONPATH=unittest .venv/bin/python unittest/test_attention_full.py -v
V8_QWEN3VL_ENCODER_PARITY_LLAMA_CPP_ROOT=./llama.cpp \
  make test-attention-f16-split-kv
make test-v8-qwen3vl
make v8-regression-fast

What This Prevents

• A correct leaf kernel hidden behind the wrong production wrapper.
• Prefill and decode silently using different accumulation contracts.
• KV=511 and KV=512 changing mathematics without a declared threshold.
• Thread count changing reduction structure without a declared partition policy.
• A model-specific Python conditional becoming permanent compiler architecture.
• A fast path being enabled because it is faster while long-token parity is worse.

Related: System Architecture, Kernel Maps and Provider Selection, IR Pipeline, v8 Vision Encoder, and Kernel Tuning Methodology.