Prefill Performance Roadmap

The current CK prefill path is correct and auditable, but the Q4_K_M gap against llama.cpp is now mostly a layout and microkernel problem.

Why Prefill Is Different

Decode is mostly GEMV: one activation vector times many weight rows. Prefill is GEMM: many token rows times many output rows. A fast decode kernel can still leave prefill slow because prefill needs to reuse both activation rows and weight blocks across a two-dimensional tile.

The key mistake is to treat prefill as many independent GEMV calls. That preserves correctness, but it reloads metadata and activation blocks too often. llama.cpp's faster path uses repacked/interleaved layouts so the hot loop computes a block of rows and columns together.

Optimization Ladder

The Q4_K Repack Target

The current CK packed-meta path is shaped like this:

for each token m:
  for each output row n:
    for each Q4_K block b:
      dot_one_q4k_block_with_one_q8k_block()

That is simple and auditable, but it is still scalar at the tile level. The next target is shaped more like this:

for token tile Mx4 or Mx8:
  pack Q8_K activations into an interleaved tile
  for output tile Nx8:
    read one interleaved Q4_K weight tile
    accumulate an MxN output tile in registers
    apply Q4_K scale/min metadata once per tile lane

This mirrors the idea behind llama.cpp's ggml_gemm_q4_K_8x8_q8_K: the math is the same, but the memory layout changes so the CPU hot loop does less bookkeeping per useful multiply. The important part is not copying llama.cpp blindly; the important part is matching the dataflow: interleave weights, interleave activations, compute a small output tile, and keep metadata close to the SIMD lanes.

CK keeps the canonical for token -> for output row -> dot() kernel as the reliable fallback. The packed-x8 lane groups eight Q4_K output rows per K block and uses a SIMD tile accumulator that loads each Q8 activation chunk once, then reuses it across the active output lanes. It is now the default for measured short/medium Qwen/Nemotron-family Q4_K prefill shapes, with CK_DISABLE_Q4K_PACKED_META_X8_PREFILL=1 as the escape hatch and CK_FORCE_Q4K_PACKED_META_X8_PREFILL=1 for new hardware sweeps.

The current x8 shape is still closer to:

for token:
  for output_row_group_of_8:
    compute 8 output rows for one Q8_K token row

The stronger target remains:

for token_tile:
  for output_tile:
    compute multiple tokens x multiple output rows together

Exact Split-Min M-Reuse Result

The promoted Qwen3-VL path does not use the older interleaved minimum-correction kernel. It keeps two FP32 accumulators per output, traverses K blocks in ascending order, performs one final value-minus-minimum subtraction, and applies bias afterward. Only independent output scheduling changes. Complete four-row groups follow the repacked GEMM contract; residual rows continue through the distinct repacked GEMV contract.

These are host-local GCC/AVX2 measurements with 20 CK worker threads and explicit hybrid-core affinity. The durable result is the provider design and exact oracle, not an expectation that every CPU will reproduce the same percentage.

The current experimental x8mt path takes the first step:

CK_ENABLE_Q4K_PACKED_META_X8MT_PREFILL=1 \
CK_Q4K_PACKED_META_X8MT_TILE_M=2 \
make profile-v8-prefill-perf-stat

In the local dispatch matrix, x8mt with tile_m=2 edged out plain x8 on the measured Q4_K shapes, including the longer M=128,N=896,K=4864 compact-down shape. It remains opt-in because end-to-end Qwen2 timings on the laptop were noisy and not yet a stable promotion signal. The next real kernel improvement is to keep the tile accumulators in registers across the token tile instead of using x8mt mostly as a scheduling/data-reuse step.

Why 2D Scheduling Was Not Enough

CK now has an opt-in Q4_K packed-meta 2D scheduler:

CK_ENABLE_Q4K_PACKED_META_2D_PREFILL=1 \
CK_FORCE_Q4K_PACKED_META_2D_PREFILL=1 \
CK_Q4K_PACKED_META_TILE_M=8 \
CK_Q4K_PACKED_META_TILE_N=512 \
CK_V8_PREFILL_PERF_ENGINE=ck \
make profile-v8-prefill-perf-stat

Local tests showed only a small and noisy improvement. That is expected: 2D scheduling creates better work distribution, but the hot loop still computes one output value at a time. The x8 layout fixed the more important part first: reuse one activation chunk across multiple output lanes. The next step is a true token-tile/output-tile microkernel.

Measurement Contract

Use the safe perf lane for routine iteration:

make profile-v8-prefill-perf-stat

The report records prefill/decode tok/s, milliseconds, cycles, instructions, IPC, cache misses, cache miss rate, and context switches. This is the default lane because VTune/Advisor deep collection can load kernel drivers that destabilized the local Linux profiling host.

Promotion Rule

1. Keep every new layout behind a shape gate and an escape hatch.
2. Prove parity against canonical CK for synthetic Q4_K/Q8_K blocks.
3. Benchmark micro-shapes: Qwen3.5 gate/up, down, qkv/out, compact MLP, and wider MLP shapes.
4. Benchmark end-to-end Qwen3.5 v8 prefill against llama.cpp with the same cached GGUF.
5. Promote only the shapes where the sweep wins; keep canonical dispatch as the fallback.

Related Files

• src/kernels/gemm_kernels_q4k_q8k_vnni.c - Q4_K x Q8_K packed-meta kernels.
• version/v8/src/ck_parallel_prefill_v8.c - v8 prefill dispatch policy.
• scripts/perf_stat_v8_prefill_compare.py - safe CK-vs-llama perf counter lane.
• benchmarks/compare_ck_llama_v8.py - standardized v8 model-family comparison.
• Q6_K Prefill Roofline Notes - shape-gating precedent for Q6_K.
• Profiling Guide - safe profiling commands and VTune/Advisor warning.