v8 Kernel Architecture

Registry at a Glance

350 provider maps, 124 logical operations. Attention dominates because every dtype × phase × layout combination is a separate physical provider with its own contract; the long tail is singleton operations such as final_logit_scale or attn_gate_softplus_mul that exist because one model family needed exactly that arithmetic.

Snapshot: version/v8/kernel_maps/KERNEL_REGISTRY.json (350 maps) + version/v8/scripts/audit_kernel_map_interfaces_v8.py at commit 9284499dc (2026-09). These counts are regenerated by hand against audit output until the docs-build generator exists.

How a Provider Is Selected

Selection is a filter chain, not a score. Compatibility gates run first; priority only ranks providers that already implement the same logical and numerical contract. Priority can never promote a candidate over a production provider, pick across equivalence groups, or override dtype, layout, phase, shape, ISA, or ABI compatibility.

The Q4_K × Q8_K AVX-512 VNNI x16 provider follows the same discipline: prefill routes are candidate behind CK_V8_FORCE_BATCHED_PREFILL=1 until 1K logit parity is certified; only the decode (m=1) prepared route is production_when_prepared.

Attention — 42 providers

One logical operation, many physical contracts. Prefill and decode are separate providers with separate phases; flash online-softmax and exact full-score attention sit in different equivalence groups because their reduction order differs. The base math (QKᵰ/√d, causal mask, softmax, online-softmax tiles) is derived in Deep Dive Concepts and Flash Attention Analysis; this section covers only the variants the registry distinguishes.

The softplus gate (attn_gate_softplus_mul_forward, hybrid_attention_kernels.c) is a per-head scalar — one gate value per head, with a linear branch above 20 for numerical safety. The sigmoid gate is per-element. They share an op shape but not a numerical contract, so they are separate providers, never an equivalence group. Laguna's softplus formula is observed from the model, not yet oracle-validated.

Multi-head latent attention (MLA)

Kimi and Instella cache a compressed latent KV vector, not per-head K/V. Decode and prefill insert explicit cache ops (mla_kv_cache_store / mla_kv_cache_batch_store), and the attention provider decompresses on the fly:

Providers: deepseek_mla_attention_f32 / _decode_f32, deepseek_mla_kv_decompress_{f32,bf16}, deepseek_mla_partial_rope_concat_{f32,packed_f32,packed_bf16_storage} — all in src/kernels/deepseek_kernels.c. Cache rows are zero-padded from head_dim to cache_stride. Full contract walkthrough: v8 MLA / Kimi decode cache.

RoPE — 18 providers + YaRN + M-RoPE

Every rotary layout is a distinct contract because the channel pairing is a checkpoint semantic, not a tuning knob. The base rotation math and the split-half vs pairwise layouts are derived in concepts — RoPE; the registry adds three more axes:

ULP-level note: the llama.cpp-exact provider (rope_precompute_cache_llama_cpu) builds the angle table by iterative FP32 multiplication (theta *= theta_scale) instead of per-pair powf, and the M-RoPE contract resolves the system libm cosf/sinf/powf via dlopen to avoid libimf divergence on ICX hosts. Bit-exactness against an oracle is a property of these details, which is why RoPE has 18 providers and not 3.

Norms — 28 providers

Quantized GEMM / GEMV — 38 providers

Weight-only quantized matmul with BF16/FP32 or Q8_K activations. Format details live in Quant Fundamentals and GEMM Memory Layout; the registry-level facts that matter for selection:

MoE — routers and experts

Four model families route: Nemotron-H (group-limited top-k + ReLU2), Kimi and Instella (sigmoid group-limited top-k + shared SwiGLU), Laguna (sigmoid + correction bias, top-8 + shared, mixed Q4/Q6 expert storage). The selection rule, verified from src/kernels/topk_kernels.c:

Recurrent / SSM — DeltaNet and Mamba2

Full derivations live in concepts — recurrent attention & Mamba2 and the Gated DeltaNet deep dive. The registry-level view:

KV Cache — persistent state, valid rows vs capacity

Cache tensors are persistent state ports, not ephemeral outputs. The layout is head-major [kv_head, token, aligned_head_dim]; capacity (max_seq_len) and valid-token count are declared separately, and kernels must never read beyond valid rows — scheduling may round to a physical extent, the kernel may not.

Fused providers — earned, not assumed

src/kernels/fused/ holds 11 fusion sources. Fusion is a measured decision, not a default: the policy (see Kernel Reference — composed vs fused) keeps operations composed in the IR unless profiling proves the fused epilogue is worth the maintenance. Registered examples: mega_fused_attention_decode_q5_0 (9 ops fused), mega_fused_attention_prefill[_q8_0] (RMSNorm → QKV → RoPE → flash → out-proj + residual), fused_mlp_block (OutProj → residual → RMSNorm → MLP → residual), fused_rmsnorm_qkv_prefill_head_major_quant, and rmsnorm_q8_k_fused (norm + activation quantization).

Logits and footer ops

Audio, vision, and training

Composite Circuits — components + stitch

Multimodal families are built by composing certified components rather than writing a monolithic new circuit. Circuits compose three ways: extends (a child deep-merges over a parent circuit, e.g. cohere_compass_text extends cohere2), components (a circuit embeds named sub-circuits, e.g. cohere_compass embeds decoder → cohere_compass_text and vision_encoder → cohere_compass_vision), and stitch edges (declared tensor handoffs between components, each carrying its own required_contract.providers). The compiler resolves extends/components recursively and fails loudly on cycles or unresolvable references.

Circuit sources: version/v8/circuits/cohere_compass.json (components + stitch), cohere_compass_text.json (extends cohere2), cohere_compass_vision.json. The same shape powers qwen36vl.json; kimi_vl.json is a flat circuit for contrast. Bring-up story: Cohere Kernel Story — North Micro Vision / Compass.

Further Reading