Prefill Performance Roadmap

The current CK prefill path is correct and auditable, but the Q4_K_M gap against llama.cpp is now mostly a layout and microkernel problem.

Why Prefill Is Different

Decode is mostly GEMV: one activation vector times many weight rows. Prefill is GEMM: many token rows times many output rows. A fast decode kernel can still leave prefill slow because prefill needs to reuse both activation rows and weight blocks across a two-dimensional tile.

The key mistake is to treat prefill as many independent GEMV calls. That preserves correctness, but it reloads metadata and activation blocks too often. llama.cpp's faster path uses repacked/interleaved layouts so the hot loop computes a block of rows and columns together.

Optimization Ladder

The Q4_K Repack Target

The current CK packed-meta path is shaped like this:

for each token m:
  for each output row n:
    for each Q4_K block b:
      dot_one_q4k_block_with_one_q8k_block()

That is simple and auditable, but it is still scalar at the tile level. The next target is shaped more like this:

for token tile Mx4 or Mx8:
  pack Q8_K activations into an interleaved tile
  for output tile Nx8:
    read one interleaved Q4_K weight tile
    accumulate an MxN output tile in registers
    apply Q4_K scale/min metadata once per tile lane

This mirrors the idea behind llama.cpp's ggml_gemm_q4_K_8x8_q8_K: the math is the same, but the memory layout changes so the CPU hot loop does less bookkeeping per useful multiply. The important part is not copying llama.cpp blindly; the important part is matching the dataflow: interleave weights, interleave activations, compute a small output tile, and keep metadata close to the SIMD lanes.

CK keeps the canonical for token -> for output row -> dot() kernel as the reliable fallback. The packed-x8 lane groups eight Q4_K output rows per K block and uses a SIMD tile accumulator that loads each Q8 activation chunk once, then reuses it across the active output lanes. It is now the default for measured short/medium Qwen/Nemotron-family Q4_K prefill shapes, with CK_DISABLE_Q4K_PACKED_META_X8_PREFILL=1 as the escape hatch and CK_FORCE_Q4K_PACKED_META_X8_PREFILL=1 for new hardware sweeps.

The current x8 shape is still closer to:

for token:
  for output_row_group_of_8:
    compute 8 output rows for one Q8_K token row

The stronger target remains:

for token_tile:
  for output_tile:
    compute multiple tokens x multiple output rows together

Exact Split-Min M-Reuse Result

The promoted Qwen3-VL path does not use the older interleaved minimum-correction kernel. It keeps two FP32 accumulators per output, traverses K blocks in ascending order, performs one final value-minus-minimum subtraction, and applies bias afterward. Only independent output scheduling changes. Complete four-row groups follow the repacked GEMM contract; residual rows continue through the distinct repacked GEMV contract.

These are host-local GCC/AVX2 measurements with 20 CK worker threads and explicit hybrid-core affinity. The durable result is the provider design and exact oracle, not an expectation that every CPU will reproduce the same percentage.

The current experimental x8mt path takes the first step:

CK_ENABLE_Q4K_PACKED_META_X8MT_PREFILL=1 \
CK_Q4K_PACKED_META_X8MT_TILE_M=2 \
make profile-v8-prefill-perf-stat

In the local dispatch matrix, x8mt with tile_m=2 edged out plain x8 on the measured Q4_K shapes, including the longer M=128,N=896,K=4864 compact-down shape. It remains opt-in because end-to-end Qwen2 timings on the laptop were noisy and not yet a stable promotion signal. The next real kernel improvement is to keep the tile accumulators in registers across the token tile instead of using x8mt mostly as a scheduling/data-reuse step.

Why 2D Scheduling Was Not Enough

CK now has an opt-in Q4_K packed-meta 2D scheduler:

CK_ENABLE_Q4K_PACKED_META_2D_PREFILL=1 \
CK_FORCE_Q4K_PACKED_META_2D_PREFILL=1 \
CK_Q4K_PACKED_META_TILE_M=8 \
CK_Q4K_PACKED_META_TILE_N=512 \
CK_V8_PREFILL_PERF_ENGINE=ck \
make profile-v8-prefill-perf-stat

Local tests showed only a small and noisy improvement. That is expected: 2D scheduling creates better work distribution, but the hot loop still computes one output value at a time. The x8 layout fixed the more important part first: reuse one activation chunk across multiple output lanes. The next step is a true token-tile/output-tile microkernel.

Exact Qwen Recurrent Q/K Normalization Parallelism

The Qwen3.5 recurrent Q/K L2 provider now partitions complete (Q-or-K, row, head) jobs through the persistent CK thread pool for sufficiently large prefill calls. Every head retains the scalar FP64 accumulation, FP32 square root, epsilon clamp, and element-update order. No reduction is split between workers. Decode, small calls, nested dispatch, and overlapping Q/K ranges continue through the serial loops.

Native tests compared one-thread and four-thread execution bit-for-bit over 257 rows, unequal Q/K head counts, and eight repetitions. The llama.cpp production oracle covered five shapes, including a 65-row case above the dispatch threshold, at 1, 16, 20, and 24 threads: all 20 comparisons passed exactly. Three baseline and three candidate full-model runs retained first-logit hash 7fedeeccedd89648 and token IDs [100, 100].

The isolated shape was rows=1024, q_dim=k_dim=2048, and head_dim=128. The combined result was measured on an Intel Core i7-14700T with AVX2/FMA and 16 physical-core representatives, using a Qwen3.5 0.8B Q4_K_M artifact with SHA-256 9297be150ac361e95e27387f5bd4b17a55a24c5428467e64bb304734f6be097d. The baseline engine SHA-256 was d865e043cf826603b1a21267d01b695e435346fe5b69c4dfd306796d1ce1e5b6; the candidate engine SHA-256 was 824c41b3c5716a70bb5a29d38ffd250583c67c2cbcc5b7ac4a7359f64c2227be. Ryzen and long-context measurements remain open.

Exact Attention-Gate Parallelism

The llama.cpp-compatible FP32 attention gate now schedules complete token rows through the persistent CK thread pool for sufficiently large prefill calls. Each output retains the scalar libm.so.6 expf call and multiplication order. Decode, small calls, nested dispatch, and unsupported overlap cases stay serial. Qwen3.5, Qwen3.8 dense, and Instella MoE consume this provider; the separately selected Qwen3.8 Flash PyTorch BF16-storage gate is unchanged.

Native tests compared one-thread and four-thread output bit-for-bit over 257 rows, 16 heads, and 128 values per head, including separate and exact in-place output across eight repetitions. A production-size 65 x 2048 comparison was bit-exact with the pinned llama.cpp oracle at 16 threads. Five baseline and five candidate Qwen3.5 full-model runs retained first-logit hash 7fedeeccedd89648 and token IDs [100, 100].

These measurements used the same Intel Core i7-14700T host and Qwen artifact described above with 1,024 repeated token IDs, two generated IDs, and context capacity 2,048. The baseline engine SHA-256 was d865e043cf826603b1a21267d01b695e435346fe5b69c4dfd306796d1ce1e5b6; the candidate engine SHA-256 was fe4c1af0514d5120038d6393363f7fa7c78c6ec2abca5196790ff4db67653cd4. Ryzen, long-context, Instella full-model, and matched llama.cpp end-to-end measurements remain open.

Measurement Contract

Use the safe perf lane for routine iteration:

make profile-v8-prefill-perf-stat

The report records prefill/decode tok/s, milliseconds, cycles, instructions, IPC, cache misses, cache miss rate, and context switches. This is the default lane because VTune/Advisor deep collection can load kernel drivers that destabilized the local Linux profiling host.

Promotion Rule

1. Keep every new layout behind a shape gate and an escape hatch.
2. Prove parity against canonical CK for synthetic Q4_K/Q8_K blocks.
3. Benchmark micro-shapes: Qwen3.5 gate/up, down, qkv/out, compact MLP, and wider MLP shapes.
4. Benchmark end-to-end Qwen3.5 v8 prefill against llama.cpp with the same cached GGUF.
5. Promote only the shapes where the sweep wins; keep canonical dispatch as the fallback.

Remaining Performance Work

The Qwen3.6 Q4 x16 prefill shapes and the exact head-parallel DeltaNet items above are completed, measured work. The next measured targets, in order, are planned work — none of them are implemented yet:

1. Q6 recurrent QKV and MLP-down short-wide GEMM providers.
2. Q8 recurrent output projection.
3. Reduction of threadpool barriers and intermediate-buffer traffic.
4. Per-layer CKE-versus-llama.cpp performance attribution.
5. Separate first-token and steady-state decode analysis.
6. Resolution of the 353-token / 1K-token numerical accumulation drift that currently keeps Qwen3.6 batched prefill opt-in.

Related Files

• src/kernels/gemm_kernels_q4k_q8k_vnni.c - Q4_K x Q8_K packed-meta kernels.
• version/v8/src/ck_parallel_prefill_v8.c - v8 prefill dispatch policy.
• scripts/perf_stat_v8_prefill_compare.py - safe CK-vs-llama perf counter lane.
• benchmarks/compare_ck_llama_v8.py - standardized v8 model-family comparison.
• v8 Runbook — Qwen3.6 optimized prefill - operator commands, dispatch verification, and current parity limitation.
• docs/notes/QWEN36_EXACT_BATCHED_PREFILL_2026-07-30.md - authoritative engineering note for PR #277.
• Q6_K Prefill Roofline Notes - shape-gating precedent for Q6_K.
• Profiling Guide - safe profiling commands and VTune/Advisor warning.