IR Pipeline v6.6

Read This Diagram in 30 Seconds

• IR1 validates template logic and resolves kernel IDs
• IR2 is where producer/consumer wiring, conservative fusion, and parallel-safe scheduling are decided
• IR3 freezes memory layout (arena offsets + pointer expressions) and emits call-ready arguments
• Runtime C executes that fixed plan using thread-pool orchestration and ISA dispatch (scalar/AVX/AVX2/VNNI)

If you forget where a bug belongs: dataflow issues usually originate in IR2, offset/alias issues in IR3, and call-order issues in codegen/runtime glue.

High-Value Test Cases and Edge Cases

• No parity runtime installed: verify expected SKIP path and explicit failure path for required parity mode
• Offline cached model flow: v6.6-validate-matrix-smoke should pass when cache is warm and network is unavailable
• Sliding-window boundaries: validate sliding_window = -1, 0, 1, > seq_len for both prefill and decode
• Model override hygiene: wrappers passing --model / --model-dir must avoid hidden default-cache behavior
• Layout compatibility: ensure flat memory.weights.entries and legacy formats both remain parseable where needed
• Concurrent gate runs: protect shared cache outputs from cross-run artifact corruption
• Long decode reliability: enforce minimum decode-run count and stable no-NaN/no-Inf summary checks

Example Template Snippet

{
  "name": "qwen3",
  "flags": {
    "use_qk_norm": true,
    "prefer_fp32_logits": false
  },
  "block_types": {
    "decoder": {
      "body": { "ops": ["attn_norm", "qkv_proj", "qk_norm", "rope_qk", "attn", "out_proj", "residual_add",
                        "ffn_norm", "mlp_gate_up", "silu_mul", "mlp_down", "residual_add"] }
    }
  }
}

IR1 builder expands this into concrete ops, assigns IDs, and attaches dataflow.

IR1 Responsibilities

• Validate template ops are mapped to kernel families
• Check kernel availability in the registry
• Attach dataflow edges (op IDs → inputs)
• Record kernel IDs without allocating memory

Implementation: version/v6.6/scripts/build_ir_v6_6.py

Example Kernel Map Snippet

{
  "id": "gemv_q8_0",
  "signature": "void gemv_q8_0(const void* w, const float* x, float* y, int m, int k)",
  "quant": { "weight": "q8_0", "activation": "fp32" },
  "buffers": {
    "w": { "role": "weight" },
    "x": { "role": "activation", "buffer": "main_stream" },
    "y": { "role": "activation", "buffer": "main_stream" }
  }
}

Example Kernel Map Entry

{
  "id": "embedding_forward_q8_0",
  "inputs": [{ "name": "tokens", "dtype": "int32", "shape": ["T"] }],
  "outputs": [{ "name": "output", "dtype": "fp32", "shape": ["T","E"] }]
}

Lowered call will pass token_ids and embedded_input pointers.

See: version/v6.6/kernel_maps/embedding_forward_q8_0.json

Do We Do QKV + RoPE "All At Once"?

Not in the default conservative v6.6 path.

• Q, K, V are produced first (three ops or a selected fused QKV op).
• RoPE runs after Q/K exist, before attention.
• Attention kernel then handles head iteration and GQA mapping internally.

A sliding-window variant uses the same contract and adds sliding_window as an extra runtime parameter.

References

• version/v6.6/kernel_maps/attention_forward_causal_head_major_gqa_flash_strided.json
• version/v6.6/kernel_maps/attention_forward_causal_head_major_gqa_flash_strided_sliding.json
• src/kernels/attention_kernels.c
• src/kernels/attention_kernels_sliding.c

IR2 Focus

• Optional fusion patterns (e.g., norm + QKV)
• Kernel variant selection for prefill vs decode
• Keeping behavior stable while enabling future speedups

IR1 Example (Stitching)

{
  "op_id": 12,
  "op": "qkv_proj",
  "outputs": { "q": "q_scratch", "k": "k_scratch", "v": "v_scratch" },
  "dataflow": { "inputs": { "x": 11 } }
}
{
  "op_id": 13,
  "op": "rope_qk",
  "dataflow": { "inputs": { "q": 12, "k": 12 } }
}

Example Generated C Call

/* Lowered IR says: gemv_q8_0(w2, x, y, m, k) */
gemv_q8_0(
    model->bump + W_L2_MLP_DOWN,   // weight pointer (from layout)
    model->layer_input,            // activation pointer
    model->mlp_scratch,            // output scratch
    INTERMEDIATE_SIZE,             // m
    EMBED_DIM                       // k
);

Codegen does not decide kernels. It emits the call with exact pointers and sizes from IR + layout.

Quick Start

python version/v6.6/tools/open_ir_visualizer.py --list
python version/v6.6/tools/open_ir_visualizer.py gemma3

# Or generate only (no browser):
python version/v6.6/tools/open_ir_visualizer.py --generate gemma3

# Generate with decode profile artifacts:
python version/v6.6/tools/open_ir_visualizer.py --generate gemma3 --with-profile --force-compile

# Generate with probes (memory sign-off + perf stat/flamegraph + perf budgets):
python version/v6.6/tools/open_ir_visualizer.py --generate gemma3 --with-probes --force-compile

This generates ir_report.html inside the model’s ck_build directory.

For cached model aliases that are not directly runnable checkpoints, pass --run-model hf://.../model.gguf. Example for Gemma: --run-model hf://unsloth/gemma-3-270m-it-GGUF/gemma-3-270m-it-Q5_K_M.gguf --chat-template none.

Manual Load

Open version/v6.6/tools/ir_visualizer.html in a browser and load:

• ir1_decode.json / ir1_prefill.json
• lowered_decode_call.json / lowered_prefill_call.json (or lowered_*.json fallback)
• layout_decode.json / layout_prefill.json
• weights_manifest.json (optional but recommended)
• profile_summary.json (optional)

Open the Generated Report

# Linux desktop
xdg-open ~/.cache/ck-engine-v6.6/models/Qwen--Qwen3-0.6B-GGUF/ir_report.html

# Or run auto-open directly (no separate xdg-open step)
python version/v6.6/tools/open_ir_visualizer.py Qwen--Qwen3-0.6B-GGUF

If your environment has no GUI, copy the generated ir_report.html to a machine with a browser and open it there.

1. Load report and select the right mode (decode or prefill).
2. Open Dataflow Graph, filter to layer/op of interest (e.g., rope_qk, attn).
3. Read the op row: find inputs with from_op and from_output.
4. Jump to that producer op ID and confirm it emits the named output with expected dtype.
5. Cross-check memory implications in Memory Layout (buffer exists, size and role are sane).

Rule of thumb: every consumer input should trace to exactly one intentional producer path. If you cannot explain the edge in one sentence, treat it as suspect.

• Decode vs prefill divergence: same op family but different kernel variant or dtype without explicit reason
• Residual path anomalies: extra/missing residual save-add pair in one layer
• Silent dtype drift: quantized weights but fp32-heavy execution path in lowered calls
• Memory planner smell: unusually large scratch growth or KV cache footprint jump
• Graph coherence: orphan node/edge patterns that suggest missing producer wiring

These are high-signal human checks that catch regressions before they become parity failures.

# Gate first
make v6.6-gate

# For release-level strictness
make v6.6-validate-parity-matrix-required

# Then inspect visual report for topology/memory sanity
python version/v6.6/tools/open_ir_visualizer.py Qwen--Qwen3-0.6B-GGUF