Whisper: Trace One Encoder Tensor into Every Decoder Layer

Linear “layer → layer” diagrams hide the interesting part of encoder-decoder models: one tensor feeding many consumers, and one operation receiving inputs from different parts of the graph. This deep dive follows Whisper's encoder memory — the tensor produced once by the encoder and consumed by every decoder layer's cross-attention — through CKE's real circuit, compiler tables, kernel maps, memory plan, and generated C. It is a companion to the model bring-up guide, which teaches the full bring-up lane; this page zooms into one tensor's edges. Machine-readable evidence: whisper_cross_attn_trace_evidence.json.

The concept: one tensor, many consumers

In a decoder-only model, the dominant dataflow pattern is a chain: layer n feeds layer n+1. Whisper's decoder breaks that pattern. The encoder runs once over the audio features and produces encoder memory E — for Whisper Base, a [1500, 512] fp32 token-major grid (1500 frames × 512 channels). Then each decoder layer applies its own projections to that one tensor:

Q_l = decoder_hidden_l x WQ_l        # from the current decoder stream
K_l = E x WK_l                       # from encoder memory, layer l's own WK
V_l = E x WV_l                       # from encoder memory, layer l's own WV
cross_attention(Q_l, K_l, V_l)

So the cross-attention operation has three input edges from different parts of the graph: Q from the decoder stream at layer l, K and V from layer l's own projections of the encoder memory. Two consequences that linear diagrams miss:

• One producer, six consumers. E is written once and read by twelve projections (K and V per layer × 6 layers). It is not the encoder's internal K/V handed down — each decoder layer owns fresh WK/WV weights and re-projects E for itself.
• The edge, not the layer, is the unit of truth. The vocabulary that survives contact with the compiler is producer (which operation writes the tensor), consumer (which operations read it), and explicit tensor edge (the declared connection between them, with dtype/shape/layout semantics). Not messages, not pub/sub — edges.

The circuit declares exactly this, in version/v8/circuits/audio_transformer_decoder.json source-verified:

"cross_attention": {
  "causal": false,
  "query_source": "decoder_stream",
  "key_value_source": "encoder_memory",
  "kv_layout": "immutable_encoder_context",
  "projection_cache": {
    "populate_phase": "prefill",
    "reuse_phase": "decode",
    "invalidation": "encoder_memory_update",
    "storage": "layer_major_head_major_fp32"
  }
}

and the decoder body names the operation instances explicitly: "cross_attn_norm", "cross_q_proj", "cross_k_proj", "cross_v_proj", "cross_attn", "cross_out_proj" between the self-attention residual and the MLP.

Who owns which fact about the tensor

A kernel map is not the whole wiring diagram. Six components each own a different fact about encoder memory; a correct bring-up writes each fact once, in the component that owns it:

Two honesty notes. First, the compiler must understand enough semantics — via contracts — to validate shapes, layouts, state, aliasing, and execution order; it does not need (or get) the handwritten arithmetic inside kernels. Second, current CKE still carries explicit operation knowledge in compiler tables: the cross_k_proj/cross_v_proj edge to external:encoder_memory is a literal entry in a dataflow table inside build_ir_v8.py (quoted below), not something derived solely from kernel maps. Do not claim otherwise.

What runs where (the honest split)

View 1 — follow the tensor

Encoder memory E from production to every consumer. The dashed middle hop is the host-side bridge (a file + np.load + one generated API call); everything to its right is the decoder's generated runtime. All shapes, sizes, and flags below come from the pinned artifact's layout_prefill.json/layout_decode.json and model_v8.c executed:

View 2 — expand one cross-attention operation

One decode-step cross-attention at layer 0, traced through every artifact layer. Click a pointer argument in the generated call to highlight the arena region it references (and, for K/V, the prefill producer edge back to encoder memory). Everything quoted is from the pinned artifact executed:

The copyable excerpts behind View 2, verbatim from the pinned artifact:

/* Op 18: attention_forward_query_key_head_major_f32_decode_heads (cross_attn) layer=0 section=body */
attention_forward_query_key_head_major_f32_decode_heads(
    (const float*)(model->bump + A_CROSS_Q_SCRATCH),     /* query <- activation:query  */
    (const float*)(model->bump + A_CROSS_K_CACHE),        /* key   <- activation:key    */
    (const float*)(model->bump + A_CROSS_V_CACHE),        /* value <- activation:value  */
    (float*)(model->bump + A_CROSS_ATTN_SCRATCH),          /* output <- output:output    */
    (float*)(model->bump + A_MLP_SCRATCH),                /* score_scratch <- scratch:* */
    8,      /* num_heads    <- dim:num_heads        */
    1,      /* query_tokens <- runtime:query_tokens */
    1500,   /* key_tokens   <- dim:key_tokens (= encoder_memory_length) */
    64,     /* head_dim     <- dim:head_dim         */
    0.125   /* scale        <- dim:attention_scale  */
);

And the layer-5 decode call reads (const float*)(model->bump + (A_CROSS_K_CACHE + 15360000)) — the same persistent cache, fifth layer slab. The K/V caches are the projected encoder memory, one slab per layer, exactly as the circuit's projection_cache.storage: layer_major_head_major_fp32 declares.

View 3 — prefill vs decode: when the projections happen

The projection cache is both a correctness story and an optimization story. Correctness: decode must never read stale projections, so installing new encoder memory invalidates the cache by flag (encoder_kv_ready = 0), and decode refuses (-2) until prefill repopulates it. Optimization: within one window, the 1500-frame encoder memory is projected once per layer at prefill and reused by every decode step — unchanged audio features are never re-projected per token. Timeline from the pinned artifact executed:

The compiler-side declaration behind this is literal and readable source-verified — build_ir_v8.py's dataflow table:

"cross_k_proj": {
    "inputs": {"x": "external:encoder_memory"},
    "outputs": {"y": {"slot": "cross_k_cache", "dtype": "fp32"}},
},
"cross_v_proj": {
    "inputs": {"x": "external:encoder_memory"},
    "outputs": {"y": {"slot": "cross_v_cache", "dtype": "fp32"}},
},

and its buffer sizing fails closed when the geometry is undeclared: "encoder-decoder artifacts require a positive encoder_memory_length" and "num_heads * head_dim == embed_dim" are hard errors, not defaults. Note again: this is explicit operation knowledge in the compiler, not knowledge derived from kernel maps.

Deliberate failure: token-major K meets a head-major consumer

Suppose a contributor wires a cross-K projection that emits token-major output straight into a cross-attention consumer whose port is head-major — and no transform is declared. Shape equality does not establish compatibility: both layouts carry the same element count, so only the declared physical layout distinguishes them. The mechanical join in version/v8/scripts/resolve_layout_chain_v8.py (circuits own logical edges; maps own physical layouts; the module joins the declarations) rejects it. Executed at this revision with the real consumer map and a synthetic token-major producer executed:

RuntimeError: no compatible physical provider chain: no physical-layout route from
cross_k_proj_token_major_hypothetical:y/token_major_contiguous/local to
attention_forward_causal_head_major_gqa_flash_compact_token_output:q/head_major_contiguous/local

Read the diagnostic as an edge report: it names the producer endpoint (producer:port/layout/placement) and the consumer endpoint — the edge is the failure, not either operation alone. The contributor's repair path is to reach the responsible contract, not to patch the kernel: declare the transform. With the real converter map registered, the same join resolves:

route: cross_k_proj_token_major_hypothetical -> layout_convert_token_to_head_f32
       -> attention_forward_causal_head_major_gqa_flash_compact_token_output | cost 10

(layout_convert_token_to_head_f32.json declares from_layout: token_major_contiguous, to_layout: head_major_contiguous, value_semantics: bit_exact_copy.) In the real Whisper circuit this transform exists as the generated transpose_cross_key_to_head_major/transpose_cross_value_to_head_major ops in prefill — the GEMM emits token-major rows, the transpose re-lays them head-major, and only then does the attention provider's port contract hold. For compiler-failure vocabulary beyond layouts (missing ports, unknown slots, aliasing without declaration), the fail-closed messages are HARD CIRCUIT INTERFACE FAULT/HARD CIRCUIT DATAFLOW FAULT from validate_circuit_interfaces_v8.py and build_ir_v8.py; the kernel-maps contributor recipe walks the general lane.

Inspect this yourself

• IR visualizer — the operations, providers, and memory offsets above render interactively, including the “Explain this operation” panel with rejection reasons: python3 version/v8/tools/open_ir_visualizer_v8.py --list (see X-Ray for the evidence model and capture-neutrality gates).
• The checkpoint named in the contract — audio.decoder.layer.{layer}.cross_attention.output (producer cross_attn, head_major) is the seam X-Ray aligns against an oracle; section 11 of the bring-up guide is the decision tree for what diverges where.
• Re-run the trace — version/v8/scripts/cks-v8-run audio hf://openai/whisper-base --wav <file.wav> --force-compile rebuilds both artifacts; the inspected run used run_whisper_v8.py run --worker-lifecycle per-window against the regenerated dirs (the wrapper's persistent-worker dispatch is environment-sensitive — see the evidence JSON note).
• Whisper operations page — Whisper Tiny on v8 covers the frontend, nightly gates, and correctness findings; this page adds the tensor-edge view.

Diagram − + ×