Qwen3-VL Numerical Parity

Qwen3-VL forced CKE to treat numerical behavior, execution schedules, cache transitions, and position semantics as explicit compiler contracts. This page records what failed, how X-ray attribution evolved, what is proven, and what remains open.

Evidence, Not A Single Parity Label

Qwen3-VL parity has several independent levels. Passing one does not imply the next.

Current Multi-Image Sweep

Top-1 equality is a behavioral gate, not a claim that every decoder logit is byte-exact. The landscape run matched all 128 selected tokens while its minimum logits cosine was approximately 0.9504. Close rankings can hide numerical drift until a later prompt or image changes the margin. For that reason the gate records cosine, RMSE, top-k overlap, margins, full replay, and the generated token trace.

Why An Ordinary Qwen3 Decoder Was Not Enough

Qwen3-VL reuses a Qwen-like text transformer, but its runtime contract is not simply vision_encoder -> ordinary_Qwen3_decode. The decoder must consume and preserve multimodal state that a text-only Qwen3 path does not have.

• The visual prefix contains projected embeddings, not token IDs that can be passed through the normal embedding lookup.
• Deepstack streams are injected at specific decoder layers in addition to the base visual embedding.
• Multimodal M-RoPE uses text, row, and column position semantics instead of ordinary one-dimensional text positions.
• Physical KV-cache indices and semantic M-RoPE positions are related but not interchangeable.
• llama.cpp executes text_before -> visual -> text_after as cache-preserving prefill segments.
• Incremental decode must append one K/V row without modifying any previous row and must advance all position state correctly.

The core decoder weights may be Qwen3-shaped, but the bridge, execution schedule, position contract, deepstack injection, and cache lifecycle make Qwen3-VL a distinct stitched circuit.

What Actually Failed

Bug Stages And Fix Ownership

The harness is part of the correctness architecture, not disposable test scaffolding. A faulty adapter can falsely accuse a correct kernel, while a permissive harness can conceal a real circuit or reduction defect. Every diagnosed harness failure therefore receives a regression test and enters the same nightly report as kernel and compiler contracts.

Rejected Hypotheses And Negative Results

A rejected experiment is retained as bounded evidence, not treated as wasted work. Its conclusion applies only to the recorded model, tensor boundary, compiler, ISA, and execution mode.

Future agents should rerun a rejected path only when at least one recorded condition changes. A new ISA, compiler, model shape, storage dtype, backend mode, or first-divergence boundary is a valid reason. Repeating the same intervention on the same evidence is not.

Why Small Differences Became Large Failures

The difficult bugs were rarely grossly incorrect matrix multiplication. They were differences in when values were rounded, how reductions were partitioned, and which state was reused. A one-ULP input change can cross a Q8 scale or code threshold. The following projection expands that changed code across many outputs, residual layers accumulate it, and a close token ranking can eventually flip even when cosine similarity remains high.

Sequence length and threading also change algorithms. At a KV threshold, a reference can move from one online-softmax reduction to worker-local partials and an ordered merge. Diagnostic capture can disable a production flash path. A test is valid only when backend mode, dtype, thread count, partitioning, runtime libraries, positions, and input bytes are recorded together. CKE's global strict flag is also an execution mode, not proof of correctness: it can select diagnostic implementations that differ from the circuit-resolved production providers.

Three Numerical Lanes

How X-Ray Had To Evolve

The first X-ray implementation compared named tensors and found the first divergent layer or op. That was necessary but insufficient for a stateful multimodal decoder. The current workflow checks execution state before expensive tensor capture.

1. Verify runtime, model, engine, shim, manifest, compiler, ISA, thread, and attention-mode identity.
2. Compare prefill segmentation and the actual kernel batch shapes.
3. Compare physical cache count, append index, strides, and semantic positions.
4. Verify the newly appended FP16 K/V row round-trips and every previous row remains unchanged.
5. Hash and compare every valid cache row, reporting the first and worst logical coordinates.
6. Use full replay at the failing token to classify persistent-state versus shared arithmetic failures.
7. Bisect sparse layers, then every layer in the failing interval, then internal op checkpoints.
8. Replay identical input bytes through the suspected quantizer or kernel before changing production math.

Capture Adapter Guardrails

• Discover checkpoint names from the exact generated decoder source paired with the runtime.
• Hard-fail unknown names, unavailable layers, or missing replay variants before model initialization.
• Capture a bounded comma-separated checkpoint set in one persistent run and one replay run.
• Build or reuse a dedicated parity-instrumented runtime; never expect a production library compiled without CK_PARITY_DUMP to emit tensors.
• Select segmented files by final physical position and reject unresolved duplicates.
• Fingerprint imported compiler, circuit, kernel-map, and schema sources so stale generated libraries cannot masquerade as current results.
• Record strict/debug, flash, repack, threading, model, engine, generated library, and manifest identity in the report.

Architecture Hardened By The Investigation

The accepted ownership model is now explicit:

• Circuits own topology, dataflow, execution phases, segmented schedules, semantic positions, required numerical contracts, and checkpoints.
• Kernel maps own exact function identity, call ABI, storage and compute dtype, rounding points, reduction order, threading arithmetic, ISA eligibility, and implementation metadata. Distinct functions cannot claim one contract unless both pass its numerical oracle.
• The resolver selects exactly one semantically compatible provider and hard-fails on zero, multiple, or incomplete matches.
• IR and code generation carry the resolved decision forward. They must not infer model families from function names or inject a model-specific fallback.
• Parity profiles own backend tensor mappings, canonical layouts, tolerances, and bounded attribution plans.

A fix belongs in the circuit when the model requires different graph or schedule semantics, in the kernel map when an existing implementation has a distinct numerical capability, and in a kernel when the required capability does not exist. The DSL should change only to express a generic missing concept, never to special-case Qwen3-VL.

Regression Gates

make test-v8-dsl
make test-v8-qwen3vl
make test-qwen3vl-methodical-parity
make test-v8-vision-kernels
make test-numerical-contracts
make test-attention-f16-split-kv
make llamacpp-parity-full

Artifact-backed encoder, mixed-prefill, teacher-forced, persistent decode, and OCR sweeps remain resource-gated. Missing models or unsupported BF16 hardware must report SKIP, not PASS. Numerical failures are hard failures; performance regressions are reported separately because machine topology and thermal state affect timing.

What Happens Next

1. Close the first OCR fixture's shared layer-0 attention/reduction mismatch with a same-input production-flash oracle.
2. Rerun the first five fixtures through at least 128 tokens and EOS where bounded.
3. Expand to ten images only after the five-image gate is clean.
4. Run the full 40-image encoder, mixed-prefill, persistent-decode, and OCR sweep.
5. Publish exact artifacts, negative results, compiler and ISA identity, and application accuracy.
6. Resume performance tuning only after correctness remains green.

Bottom Line

Qwen3-VL was difficult because correctness spans more than kernels and more than a tensor graph. It spans image geometry, dtype boundaries, evaluation order, execution schedules, position spaces, cache transitions, backend modes, and persistent state. X-ray made those contracts observable and converted repeated manual debugging into a bounded first-divergence process. The encoder evidence is now strong, but the project will not call end-to-end parity complete until the varied-image persistent-decode and OCR gates pass.