Kernel Maps and Provider Selection

How CKE v8 decides which physical C kernel executes each logical operation — and why that decision lives in 284 machine-checked JSON maps instead of in Python conditionals.

The Three-Level Separation — and Why

One CKE runtime serves many model families × quantization formats × ISA variants × execution phases, all on CPUs. That product space is why v8 separates what math is required from which provider runs from how the op is physically executed:

Anatomy of a Kernel Map

version/v8/kernel_maps/memcpy.json is the canonical annotated example — the production map behind the residual_save op. Every field group has one job:

Selection metadata is schema-enforced

The selection block is required by version/v8/schemas/kernel_provider_selection.schema.json and carries exactly four fields:

The hard rules: an equal-priority tie between explicit production providers is a fault, not a coin flip. And the map — not the code generator — owns the call ABI: dims, typed params (here _memcpy_bytes of type size_t), and a versioned call_abi block naming the exact argument list (dst, src, size) with each argument's source port. The alias and arithmetic contract is part of the map too: constraints.notes states this is a byte copy, performs no arithmetic, and is undefined for overlapping writable views.

Reading a Kernel Map: Field Reference

The anatomy figure above shows the shape of one map. This section is the member-by-member reference: what each field means, which component consumes it, and what happens when it is missing or wrong.

Worked Example 1: memcpy — the Baseline Read

memcpy.json is the minimal production map: one op, two ports, one computed parameter, no ISA variants, no numerical contract — a byte copy has no arithmetic to contract. Verbatim excerpt (… marks elided lines):

{
  "id": "memcpy",
  "op": "residual_save",
  "operation_interface": "residual_save.memcpy_copy.v1",
  "selection": {"status": "production", "priority": 100, "equivalence_group": "residual_save.memcpy_copy.v1", "phases": ["prefill", "decode"]},
  "variant": "fp32_copy",
  "quant": {"weight": null, "activation": "fp32", "output": "fp32"},
  "inputs":  [{"name": "src", "dtype": "fp32", "shape": ["N"], "layout": "contiguous", "access": "read",  "storage_class": "activation", …}],
  "outputs": [{"name": "dst", "dtype": "fp32", "shape": ["N"], "layout": "contiguous", "access": "write", "storage_class": "activation", …}],
  "dims": ["N"],
  "params": [{"name": "_memcpy_bytes", "type": "size_t", "desc": "Number of bytes to copy"}],
  "constraints": {"notes": "Byte copy implementing logical residual_save; performs no arithmetic. src and dst must not overlap …"},
  "impl": {"function": "memcpy", "sources": [], "variants": [{"name": "default", "requires": [], "compile_flags": []}]},
  "call_abi": {"version": 1, "params": [
    {"name": "dst",  "source": "output:dst",        "cast": "void*", …},
    {"name": "src",  "source": "activation:src",    "cast": "const void*", …},
    {"name": "size", "source": "dim:_memcpy_bytes"} ]},
  "tests": {}
}

How to read it. The resolver reaches this map when a circuit needs residual_save in prefill or decode: the selection block makes it eligible in both phases, quant matches FP32 activations, and the single default variant passes ISA filtering vacuously (requires: []). Codegen then emits exactly memcpy(dst, src, size) — no argument reconstruction. What it deliberately lacks: numerical_contract, numerical_capabilities, non-empty tests. The audit therefore classes it as selection-managed and interface-ready, not fully "hardened" — a legitimate state for a copy with one possible behavior.

Worked Example 2: gemm_nt_q8_0_q8_0 — the Rich One

The Q8 prefill GEMM shows every group memcpy omits: a named numerical contract, ISA-gated variants, map-owned ABI for seven arguments, and validation oracles.

{
  "id": "gemm_nt_q8_0_q8_0",
  "op": "gemm",
  "selection": {"status": "production", "priority": 200,
                "equivalence_group": "q8_0_weight_q8_0_input_llama_fp32_output",
                "phases": ["prefill"]},
  "numerical_contract": "q8_0_weight_q8_0_input_llama_fp32_output",
  "quant": {"weight": "q8_0", "activation": "q8_0", "output": "fp32"},
  "inputs": [
    {"name": "A", "dtype": "q8_0", "shape": ["M", "K"], "layout": "token_major_contiguous", …},
    {"name": "B", "dtype": "q8_0", "shape": ["N", "K"], "layout": "opaque_packed", …},
    {"name": "bias", "dtype": "fp32", "shape": ["N"], …, "optional": true, …} ],
  "outputs": [{"name": "C", "dtype": "fp32", "shape": ["M", "N"], "layout": "token_major_contiguous", …}],
  "dims": ["M", "N", "K"],
  "constraints": {"alignment": {"K": 32}, "notes": "Q8_0 requires K dimension multiple of 32 …"},
  "phase_selection": {"decode": {"op_family": "gemv", "default_kernel": "gemv_q8_0_q8_0",
                                 "notes": "Single-token decode retains the certified GEMV path."}, …},
  "impl": {"function": "gemm_nt_q8_0_q8_0", "sources": ["src/kernels/gemm_batch_int8.c", …],
    "variants": [
      {"name": "avx2_m2n4", "function": "gemm_nt_q8_0_q8_0_m2n4", "priority": 240,
       "requires": ["avx2", "fma"], "shape_constraints": {"M_min": 2, "K_multiple": 32},
       "numerical_contract": "q8_0_weight_q8_0_input_llama_fp32_output"},
      {"name": "avx2",     "requires": ["avx2", "fma"],              "compile_flags": ["-mavx2", "-mfma"]},
      {"name": "avx_vnni", "requires": ["avx2", "avx_vnni", "fma"],  "compile_flags": ["-mavx2", "-mavxvnni", "-mfma"]} ]},
  "reference": {"function": "vec_dot_q8_0_q8_0_ref", "kind": "scalar_contract_oracle", …},
  "production": {"function": "gemm_nt_q8_0_q8_0", …, "reference_comparison": {"requirement": "bit_exact", …}},
  "call_abi": {"version": 1, "params": [
    {"name": "A", "source": "activation:a", "cast": "const void*"},
    {"name": "B", "source": "weight:_first_weight"},
    {"name": "bias", "source": "weight_f:_bias"},
    {"name": "C", "source": "output:c", "cast": "float*"},
    {"name": "M", "source": "dim:_m"}, {"name": "N", "source": "dim:_output_dim"},
    {"name": "K", "source": "dim:_input_dim"} ]},
  …
}

How the resolver uses it. Three facts interact:

• The equivalence-group name encodes the arithmetic. equivalence_group and numerical_contract are the same string — q8_0_weight_q8_0_input_llama_fp32_output: Q8_0 weights times Q8_0 activations producing FP32 output under llama.cpp-compatible semantics. Because priority ranks only within a group, no priority value can ever promote a provider with different arithmetic over this one; cross-group ties are a hard fault ("priority cannot choose between different equivalence groups"), not a silent substitution.
• Phases filter before priority ranks. phases: ["prefill"] means a decode op never sees this provider at all — phase_selection.decode documents that single-token decode keeps the certified gemv_q8_0_q8_0 GEMV path instead. Priority 200 is only ever compared against other prefill Q8 GEMM providers in the same group.
• Variants carry ISA eligibility. Each variant's requires list gates it by CPU features: on a machine without AVX-VNNI the avx_vnni variant is ineligible and filtered out. The avx2_m2n4 tile variant adds its own priority: 240 and shape constraints (M ≥ 2, K multiple of 32) — variant-level preference, still inside the same numerical contract, which it repeats verbatim.

Honesty note: this map has no operation_interface and no numerical_capabilities block, so the audit classes it as selection-managed and legacy-contract-shaped — a production provider carrying the older contract keys, exactly the kind of map the migration ratchet tracks.

Worked Example 3: kv_cache_store_bf16 — the Persistent-State One

Cache stores look like ordinary ops but their outputs are state, not ephemeral tensors. Verbatim excerpt:

{
  "id": "kv_cache_store_bf16",
  "op": "kv_cache_store",
  "operation_interface": "kv_cache_store.bf16.head_major.v1",
  "selection": {"status": "production", "priority": 100,
                "equivalence_group": "kv_cache_store.bf16.head_major.v1", "phases": ["decode"]},
  "variant": "bf16_cache",
  "quant": {"activation": "fp32", "output": "bf16"},
  "inputs": [
    {"name": "k", "dtype": "fp32", "shape": ["KV", "D"], "layout": "head_major_contiguous",
     "access": "read", "storage_class": "activation", "consumption": "required", …},
    {"name": "v", …} ],
  "outputs": [
    {"name": "kv_cache_k", "dtype": "bf16", "shape": ["KV", "S_max", "D"], "layout": "head_major_contiguous",
     "access": "read_write", "storage_class": "state", "consumption": "required",
     "desc": "Packed BF16 K cache slice at current pos. Physical capacity S_max rows; valid-token count is tracked separately by the caller (append index); …"},
    {"name": "kv_cache_v", …} ],
  "dims": ["num_kv_heads", "head_dim", "num_layers", "max_seq_len", "KV", "S_max", "D"],
  "constraints": {"notes": "Scheduling may round the extent to physical capacity S_max; kernels must never read or write KV rows beyond the valid-token count declared by the append index."},
  "impl": {"function": "kv_cache_store_bf16", …, "sources": ["src/kernels/kv_cache_kernels.c"]},
  "call_abi": {"version": 1, "params": [
    {"name": "kv_cache_k", "source": "runtime:kv_cache_k_layer_u16", "cast": "uint16_t*", …},
    {"name": "kv_cache_v", "source": "runtime:kv_cache_v_layer_u16", "cast": "uint16_t*", …},
    {"name": "k", "source": "activation:k", "cast": "const float*", …},
    {"name": "v", "source": "activation:v", "cast": "const float*", …},
    {"name": "layer", "source": "const:0"}, {"name": "pos", "source": "runtime:pos"},
    {"name": "num_kv_heads", "source": "dim:num_kv_heads"},
    {"name": "head_dim", "source": "dim:head_dim"}, {"name": "max_seq_len", "source": "dim:max_seq_len"} ]},
  …
}

How to read it.

• State, not output. The kv_cache_k/kv_cache_v ports are storage_class: "state" with access: "read_write" — the kernel appends into a persistent cache slice rather than producing a fresh tensor. That classification is what lets the memory planner treat the cache as long-lived state instead of a per-op allocation.
• Capacity vs validity. The shape ["KV", "S_max", "D"] is physical capacity: S_max rows exist, but the valid-token count is tracked separately by the caller through the append index (pos in the call ABI, sourced from runtime:pos). The constraint is explicit: append writes exactly the rows at the append index and never reads beyond valid rows, even if scheduling rounds extents up to S_max.
• Storage dtype and layout are part of the contract. The interface and equivalence group — kv_cache_store.bf16.head_major.v1 — bake in BF16 storage and head-major layout. An FP16 or FP32 cache store is a different group, so the resolver can never swap storage formats as a "priority" decision; quant says FP32 in, BF16 stored.
• Decode-only is deliberate. phases: ["decode"] keeps this provider out of prefill entirely; prefill KV writes use their own providers. Like memcpy it carries operation_interface but no numerical_capabilities, so the audit classes it as selection-managed and legacy-interface-ready.

With the format and three real maps in hand, the resolution algorithm below is mechanical: everything the resolver knows about a provider comes from these fields, and everything it rejects leaves a recorded reason.

The Resolution Algorithm

Provider selection lives in _provider_selection_metadata and _rank_provider_matches (build_ir_v8.py:5666-5770). The pipeline:

1. Validate selection metadata. Every provider must declare an equivalence_group; a missing or malformed selection block is a HARD KERNEL SELECTION FAULT. Providers with no selection block at all are treated as legacy.
2. Filter by compatibility. Contract identity, phase, dtype, shape, layout, ISA, and alias safety — any mismatch rejects the provider with a recorded reason.
3. Rank the survivors. The ranking tuple is direction (inference vs backward) → activation preference → lifecycle rank (production first) → priority (higher wins, within the group).
4. Fail closed on ambiguity. A tie among explicit production providers raises "equal-priority production providers are ambiguous" when they share a group, or "priority cannot choose between different equivalence groups" when they do not. Zero compatible providers is also a compile-time failure — the build never silently substitutes different numerics.

Evidence, not assertion. The X-Ray fixture version/v8/tests/fixtures/xray/provider_selection_trace.json records a real selection trace: a candidate provider with priority 900 is rejected for status_not_production, a priority-800 production provider for phase_mismatch, and a priority-700 production provider for weight_dtype_mismatch — all before the compatible priority-100 production provider is selected at rank 0. Higher priority loses to incompatibility every time, because compatibility filtering happens before priority ranking.

Case Study: residual_save → memcpy

This is the headline example of DSL logic that legitimately remains — and of what hardened selection replaced. The two concerns are different, and the code now keeps them in different places:

• Insertion is scheduling, and stays in the DSL. should_insert_residual_save (build_ir_v8.py:3743) fires when the current op is a pre-norm, the next op starts a branch (attention or MLP), and the previous op is not already a residual save. The op is auto-inserted and stamped "kernel": "memcpy" at build_ir_v8.py:7328; the _memcpy_bytes buffer size (embed_dim × seq_len × 4) is computed during lowering at build_ir_v8.py:8610. Deciding that a residual must be saved is graph scheduling — it belongs here.
• Selection is hardened. The memcpy map is selection-managed with declared copy semantics — a byte copy, not residual arithmetic mislabeled under a copy's name. It wins or loses through the same mechanical resolver as every other hardened provider.
• The debt is named, not hidden. map_op_to_kernel (build_ir_v8.py:6712) still carries template-override branches: rope_qk via _resolve_rope_qk_kernel, rope_q, mrope_qk, position_embeddings via _resolve_position_embeddings_kernel, kv_cache_store_shared_q, and the assistant_layer_scale override. Separately, _make_decode_kv_store_op (build_ir_v8.py:8782) fabricates decode KV store ops. This is migration debt being actively burned down — tracked by the ratchet below.

Where the Weights Side Fits

Provider selection decides which kernel runs; the weights pipeline decides which bytes it reads. Converted BUMP weights carry a metadata sidecar describing tensor names, dtypes, and layouts (see GGUF to Bump and ADR 0006). Circuits and templates reference weight tensors, and the map's quant block and weights ports must match what the sidecar describes — a map that expects Q4_K weights will not pass compatibility filtering against an FP32 tensor. The conversion format itself is documented on the BUMP page; it is not re-explained here.

Evolution Timeline

An honest, PR-dated history. The direction is monotonic: physics moves into the maps, heuristics leave the DSL.

• Legacy: kernel_bindings.json and a function-name ABI fallback — the name was the contract.
• PR #302: hardened provider selection (lifecycle, priority, equivalence groups) and the operation-interface format introduced.
• PR #305: shared RoPE, residual-copy, and KV-cache providers migrated to selection-managed maps.
• PR #318 / #320: layout provider selection and direct-layout attention providers — physical layout joins the map-owned contract.
• Future: resolver cleanup (rebind circuits to hardened metadata, delete the legacy branches), then attention and quantized GEMM migrations. The end state is an agnostic DSL: circuits state contracts, maps own everything physical, the resolver contains no model knowledge at all.

Migration Scoreboard

version/v8/contracts/kernel_interface_migration_baseline.json is a monotonic ratchet checked by version/v8/scripts/audit_kernel_map_interfaces_v8.py: floors may only rise, ceilings may only fall. The audit currently reports the baseline values exactly — this is the starting line the burn-down is measured from:

Is This Over-Engineered?

Fair question — the project owner has asked it directly. 284 JSON maps, a schema for four fields, a ratchet file, and an audit script can look like architecture for its own sake. The honest answer has three parts.

Related: v8 Numerical Contracts, System Architecture, Code Generation, GGUF to Bump, and Architecture Links.