--- layout: default title: Q6_K Prefill Roofline Notes permalink: /q6-prefill-roofline/ ---

Q6_K Prefill Roofline Notes

Why the Q6_K x Q8_K 2D prefill scheduler is shape-gated instead of enabled globally.

Summary: the 2D scheduler is a load-balancing optimization, not a replacement for row-split prefill. It helps wide Q6_K shapes such as Nanbeige MLP-down, but can regress narrower Qwen2/Gemma/Qwen3.5 shapes.

Kernel Under Test

The benchmark targets gemm_nt_q6_k_q8_k_parallel_dispatch() in version/v8/src/ck_parallel_prefill_v8.c. The math kernel computes C = A @ B^T, where A is token-major Q8_K activations and B is Q6_K weights. This is used by v8 prefill for model-family MLP-down and some projection/logit paths.

Roofline Interpretation

Row-split scheduling assigns token rows to workers. For each worker, a Q8_K activation row is reused across the full output dimension N. The 2D scheduler splits both token rows and output columns. That creates more jobs and better load balance, but rereads the same Q8_K activation tile once per N tile.

On memory-bound Q6_K prefill, the extra Q8_K activation traffic is only worth it when the shape is wide enough that row-split does not keep cores busy or does not balance work well. Narrow output dimensions usually stay faster with row splitting.

Local Measurement

Measurements below were taken on the local i7 development machine on 2026-06-09 with CK_NUM_THREADS=12, OMP_NUM_THREADS=1, and CK_Q6K_Q8K_SIMD=1. The benchmark used synthetic finite Q6_K/Q8_K blocks and compared row-split dispatch against raw 2D dispatch with CK_FORCE_Q6K_Q8K_2D_PREFILL=1.

Shape	M	Row Split	2D Tile	Result
Qwen2-like N=896 K=4864	16	0.97 ms	2.02 ms	2D slower
Qwen2-like N=896 K=4864	64	2.82 ms	3.05 ms	2D slower
Qwen2-like N=896 K=4864	256	9.89 ms	9.94 ms	neutral/slower
Qwen2-like N=896 K=4864	512	19.27 ms	19.00 ms	2D +1.4%
Large Q6 N=2560 K=10240	16	5.80 ms	4.46 ms	2D +23.1%
Large Q6 N=2560 K=10240	128	33.09 ms	28.92 ms	2D +12.6%
Large Q6 N=2560 K=10240	1024	253.92 ms	233.13 ms	2D +8.2%

Family-Level Signal

A longer prompt smoke test across cached v8 families showed the same pattern: Qwen2 regressed with raw 2D enabled, Qwen3.5 was slightly slower, Gemma was neutral, and Nanbeige improved modestly. Qwen3 in the local cache was Q8-heavy, so this Q6-specific path was mostly neutral.

Dispatch Policy

CK_ENABLE_Q6K_Q8K_2D_PREFILL=1 remains opt-in, but the dispatcher now also checks shape. By default, 2D Q6 prefill is selected only for wide shapes with N >= 2048 and K >= 8192, with enough tile jobs to keep the active thread pool busy.

For raw benchmarking, use CK_FORCE_Q6K_Q8K_2D_PREFILL=1. Do not use the force flag in production or model-family regression runs unless the goal is to measure the scheduler itself.

Deployment Tuning Model

The long-term deployment flow should treat kernel selection as a hardware/model tuning artifact:

Build or load the model runtime and extract the lowered kernel shapes.
Run a bounded sweep for the hardware with --sweep-kernels.
Write the result to the runtime directory as kernel_tuning.json.
Have IR/codegen or dispatch read that file on later runs.
If the file is missing, use conservative built-in defaults.

This keeps production deterministic while still allowing each CPU target to choose the best kernel policy for its cache, memory bandwidth, ISA, and thread topology. The current Q6 sweep runner is the first concrete version of that pattern.

python -B version/v8/scripts/ck_run_v8.py run build/regression-runs/qwen2 \
  --run build/qwen2-tuned --context-len 1024 --generate-only --sweep-kernels

python -B version/v8/scripts/ck_run_v8.py run build/regression-runs/qwen2 \
  --run build/qwen2-tuned --context-len 1024 --generate-only --sweep-kernels --sweep-kernels-full

Reproduction

make build/libckernel_engine.so
make test-q6k-prefill-dispatch-sweep-quick
make test-q6k-prefill-dispatch-sweep
make test-q6k-prefill-dispatch-sweep-avx2

CK_NUM_THREADS=12 OMP_NUM_THREADS=1 CK_Q6K_Q8K_SIMD=1 \
  python -B benchmarks/bench_q6k_prefill_tile.py \
  --mode compare --m 1024 --n 2560 --k 10240 --threads 12 --warmup 1 --iters 3

CK_FORCE_Q6K_Q8K_2D_PREFILL=1 CK_ENABLE_Q6K_Q8K_2D_PREFILL=1 \
CK_NUM_THREADS=12 OMP_NUM_THREADS=1 CK_Q6K_Q8K_SIMD=1 \
  python -B benchmarks/bench_q6k_prefill_tile.py \
  --mode 2d --m 128 --n 2560 --k 10240 --threads 12 --warmup 1 --iters 2