Stitched Divergence Harness

How C-Kernel-Engine finds the first numerical divergence between generated C, llama.cpp/mtmd, and future PyTorch adapters without guessing from end-to-end output.

Core rule: the harness does not dynamically understand every backend by itself. It compares backends through explicit adapters that dump the same named tensors, at the same layer/op boundaries, in the same logical layout.

Why This Exists

Kernel unit tests prove that an isolated function can be correct. They do not prove that a full model circuit is stitched correctly. A model can still fail because of tensor layout, positional semantics, weight interpretation, preprocessing, quantized activation contracts, residual timing, or a dump boundary mismatch.

The stitched divergence harness answers a narrower and more useful question: where is the first boundary where CK stops matching the reference? Once that boundary is known, the fix can target one kernel, one conversion rule, one transpose, or one template seam instead of the whole transformer.

Current Backend Contract

Backend	Current role	How tensors are obtained	Status
`CK generated C`	Test implementation	Generated code emits `ck_dump_tensor(...)` records when built with parity dumping.	Implemented for v8 text and Qwen3-VL vision paths.
`llama.cpp + mtmd`	GGUF reference for Qwen3-VL and quantized inference	A local C++ shim links against llama.cpp/mtmd and writes selected ggml tensors into CKDMP format.	Implemented for the current Qwen3-VL GGUF lane.
`llama.cpp decoder helper`	Reference for token replay and logits	Persistent greedy replay and full replay compare CK logits/top-k against llama.cpp.	Implemented for v8 multimodal decode parity.
`PyTorch / safetensors`	BF16/full-precision reference lane	Requires a PyTorch adapter that emits the same named tensors and layouts as CKDMP.	Available in model-specific scripts, not yet the default backend for stitched Qwen3-VL GGUF parity.

Do not mix reference lanes casually. For Qwen3-VL AVX2 GGUF parity, enforce llama.cpp/mtmd. For BF16 safetensors circuit bring-up, use PyTorch. The same harness structure can support both, but only when each backend exposes equivalent tensor boundaries.

Does CK Patch llama.cpp?

The current Qwen3-VL GGUF lane does not rewrite llama.cpp at test time. CK builds a small local adapter: version/v8/tools/mtmd_clip_shim.cpp. That shim includes the local llama.cpp/mtmd headers, initializes the mtmd clip context, registers a ggml evaluation callback, and writes selected tensors to the same CKDMP dump format used by generated CK code.

This is deliberately different from hard-forking the reference. The reference remains llama.cpp's graph and weights. CK only adds an observation surface around it. If upstream llama.cpp changes its private mtmd APIs or removes the callback seam, the adapter must be updated, or the local llama.cpp checkout must carry the minimal callback support needed for parity dumps.

Dump Format

Both CK and reference adapters write dump.bin using the CKDMP record format from version/v8/src/ck_parity_dump.h:

128-byte record header
layer_id
op_name
dtype and rank metadata
shape and element count
token id for decode paths
flat tensor payload, usually converted to float32 for comparison

The comparison layer matches records by normalized operation name, layer, shape, and candidate order. Alias handling maps names such as attn_output to llama-side names such as attn_out when the two backends use different labels for the same graph boundary.

How CK Dumps Tensors

CK is easier to instrument because the model runtime is generated C. The v8 codegen path can emit dump calls at known IR boundaries:

ck_dump_tensor((float*)buffer, layer_id, "ffn_up_b", element_count);
ck_dump_tensor_head_major_token_major(q_buffer, layer_id, "Qcur", heads, tokens, head_dim);

Environment variables constrain the dump so a full 8B model does not fill scratch:

CK_PARITY_DIR selects the CK dump directory.
CK_PARITY_LAYER_FILTER restricts layers.
CK_PARITY_OP_FILTER restricts operation names.
--granular-test emits dump/cutpoint metadata during codegen.
--ck-stop-layer or --ck-stop-op returns early after a target boundary.

How llama.cpp / mtmd Dumps Tensors

The llama.cpp side is adapter-driven. The Qwen3-VL mmproj adapter compiles mtmd_clip_shim.cpp into a shared object, loads llama.cpp libmtmd.so, and calls clip_encode_float_image. The shim's callback receives ggml tensors as llama.cpp evaluates the graph.

The shim uses these controls:

CK_LLAMA_PARITY_DIR writes llama's dump.bin.
CK_LLAMA_PARITY_NAMES selects tensors such as Qcur, kqv_out, or attn_out.
CK_LLAMA_PARITY_LAYER restricts to one layer.
CK_LLAMA_PARITY_ALL=1 dumps every visible tensor, used only for deep diagnostics.
CK_LLAMA_PARITY_META=1 writes tensor shape/stride metadata as JSONL.

Quantized, fp16, bf16, and integer tensors are flattened into a comparable fp32 payload. That means the dump compares the logical value at the selected boundary, not necessarily the raw packed bytes. Raw packed-byte checks are separate tests for weight conversion and quant block contracts.

How PyTorch Fits

PyTorch is not automatically interchangeable with llama.cpp. It can become a stitched backend only when it implements the same adapter contract:

Run the same canonical input and prompt.
Register hooks or use a traced graph to capture named layer/op boundaries.
Normalize layout to CK's logical comparison layout.
Write CKDMP or a format converted losslessly into CKDMP.
Use the same alias map and thresholds as the CK-vs-reference comparator.

For BF16/safetensors bring-up, PyTorch is the better reference because it removes GGUF quantization and llama.cpp packing decisions from the equation. For GGUF deployment parity, llama.cpp is the stronger reference because it exercises the same file format, quantized tensor semantics, image path, and token replay behavior expected in production.

Stitched Runner Flow

version/v8/scripts/stitched_parity_v8.py runs stages from cheap and broad to expensive and granular:

Bridge generation. CK builds the image/text bridge and writes bridge_report.json plus prefix.f32.
Encoder numeric parity. CK's final vision prefix is compared against llama.cpp/mtmd output. If this fails, decoder debugging stops because the prefix is already wrong.
Persistent multi-token parity. If the prefix is acceptable, CK and llama.cpp run greedy token replay. The report records first mismatch step, token ids, top-k overlap, cosine, RMSE, and logits metrics.
Granular attribution. If a coarse stage fails, the runner dumps selected layer/op tensors and reports the first failing boundary.

.venv/bin/python version/v8/scripts/stitched_parity_v8.py \
  --template qwen3vl \
  --mode fast \
  --decoder-gguf models/Qwen3-VL-8B-Instruct-GGUF/Qwen3VL-8B-Instruct-Q4_K_M.gguf \
  --mmproj-gguf models/Qwen3-VL-8B-Instruct-GGUF/mmproj-Qwen3VL-8B-Instruct-Q8_0.gguf \
  --image-path build/qwen3vl_avx2_fresh_bridge_20260708/1_81_canonical.ppm \
  --prompt "Extract visible form fields as compact JSON." \
  --workdir build/stitched_parity/qwen3vl_avx2_canonical \
  --ctx-len 4096 \
  --threads 20 \
  --top-k 16 \
  --max-new-tokens 64 \
  --image-max-tokens 1024 \
  --phase-timeout-sec 1800 \
  --log-byte-limit 1048576

How It Decides Where To Break

The system does not infer a graph cut from arbitrary source code. The cutpoints come from the generated IR/codegen boundary map and backend-specific dump names. For Qwen3-VL vision, the useful layer boundaries include:

inp_pos_emb
patch_bias
ln1
Qcur, Kcur, Vcur
Qcur_rope, Kcur_rope
kqv_out
attn_out / attn_output
ffn_inp, ffn_inp_normed
ffn_up_b, ffn_out
layer_out

A failure like kqv_out PASS followed by attn_output FAIL immediately narrows the search to the output-projection boundary, transpose/layout before projection, activation quantization, weight view, residual add, or a naming mismatch around that exact dump.

Report Semantics

A stitched report is designed to be pasted into a PR or agent handoff. Important fields are:

status: pass/fail/timeout.
failure_stage: bridge, encoder_numeric, multitoken, or granular.
encoder_numeric_metrics: cosine, RMSE, max abs, mean abs.
first_mismatch: generated token step and CK/llama token ids.
first_granular_issue: layer, op, diff metrics, divergent index.
commands.log: exact subprocess commands and trimmed output.

Adding A New Model Or Backend

New compatibility work should add the following pieces before optimizing kernels:

Canonical run spec: model files, checksums, prompt, context length, thread count, image/audio preprocessing, and deterministic sampling settings.
CK dump map: generated code emits named tensors at stable IR boundaries.
Reference adapter: llama.cpp, PyTorch, or another backend emits the same boundaries in CKDMP-compatible form.
Alias map: normalize names such as attn_output vs attn_out, and document any boundary that is not semantically identical.
Layout contract: define whether the comparison is token-major, head-major, row-major, packed, or unpacked logical fp32.
Thresholds: strict where the math should be exact, looser only where the reference intentionally differs by dtype or quantization.
Fallback triage: if full output fails, rerun layer 0, then layer range, then op-specific dumps.

Known Failure Modes

Duplicate Tensor Names

If CK dumps both pre-projection and post-projection tensors as attn_output, the comparator may select the wrong candidate. Use unique names such as attn_context, out_proj_in, and attn_output.

Layout Mismatch

CK often stores attention scratch in head-major layout while references expose token-major tensors. Dump helpers must normalize logical order before comparison.

Reference Boundary Mismatch

A name match is not enough. Verify whether the reference tensor is before bias, after bias, before residual, after residual, before activation, or after activation.

Scratch Explosion

Full-model dumps are enormous. Use layer filters, op filters, early CK stop, log byte caps, and weight-pruning after each phase.

Operational Guidance

For AVX2 GGUF Qwen3-VL: use llama.cpp/mtmd as the enforced backend.
For BF16 safetensors template bring-up: use PyTorch as the reference backend.
Do not optimize a fast path until stitched parity identifies the first failing boundary.
Do not relax tolerances to hide a top-1 flip or long-decode drift.
When a kernel unit test passes but stitched parity fails, suspect layout, conversion, or graph semantics before rewriting the kernel.
When a reference adapter changes, record the llama.cpp commit or PyTorch model revision in the report.

What good looks like: a failed run should say "layer 0, op attn_output, kqv_out passed, output projection boundary failed", not merely "OCR answer is wrong". That is the difference between model debugging and guessing.