Deep Dive: LLM Concepts

This page explains the key concepts, algorithms, and computational techniques used in modern Large Language Models. Understanding these fundamentals helps you know why each kernel exists and how they fit together.

Table of Contents

• Transformer Architecture Overview
• RoPE: Rotary Position Embedding
• RoPE Layouts: Pairwise vs Split-Half
• GLM-4 RoPE: Pairwise Partial Rotary
• Gemma4 RoPE: Per-Layer Direct Split-Half
• Attention Mechanisms
• Flash Attention
• Recurrent Attention and Gated DeltaNet
• Sliding-Window Attention
• Grouped Query Attention (GQA)
• Sparse / Query-Selected Attention (QSA)
• Normalization: RMSNorm vs LayerNorm
• Nemotron-H Hybrid Circuit
• Mamba2 Reference Kernels
• Mixture of Experts: Router + ReLU2 Experts
• Activations: ReLU2, SwiGLU, GeGLU, GELU
• Vision Patch Embedding
• — 1D Text vs 2D Vision Position (RoPE)
• Audio Frontend: WAV to Log-Mel to Conv Stem
• — Cross-Attention: Decoder Queries, Encoder Memory
• Weight Tying
• Hyper-Connections (Multi-Stream Residuals)
• Per-Layer Embedding (PLE)

Transformer Architecture Overview

Before diving into individual components, let's see where everything fits in a decoder-only transformer (like Llama, GPT).

RoPE: Rotary Position Embedding

RoPE encodes position information by rotating query and key vectors based on their position in the sequence. Unlike absolute position embeddings, RoPE naturally captures relative position.

Traditional (GPT-2 style):
  token_embed + position_embed → layers

With RoPE:
  token_embed → layers (RoPE applied inside attention)

Token: "The"  "cat"  "sat"  "on"   "mat"
Pos:     0      1      2      3      4

Embedding lookup (NO position added):
  [vec0] [vec1] [vec2] [vec3] [vec4]
           |
      Decoder Layer
           |
      Q, K, V projection
           |
    +---------------------------+
    |  RoPE: rotate Q and K     |
    |  Q_0 rotated by 0*theta   |
    |  Q_1 rotated by 1*theta   |
    |  Q_2 rotated by 2*theta   |
    |  (same angles for K)      |
    +---------------------------+
           |
      Attention: Q @ K.T (now position-aware!)
    

theta_i = rope_theta ^ (-2i / d)

For d=64, i=0:  theta = 10000^0 = 1
For d=64, i=16: theta = 10000^(-0.5) = 0.01
For d=64, i=31: theta = 10000^(-0.97) ~ 0.0001

angle(pos, dim) = position * theta_i

At position m:
  Q_m = rotate(Q_m, angle = m * theta)
  K_m = rotate(K_m, angle = m * theta)

At position n:
  K_n = rotate(K_n, angle = n * theta)

Q_m · K_n = f(Q, K, m - n)

Forward:
    theta_i = rope_theta ^ (-2i / head_dim)
    angle = m * theta_i

    x'[2i]   = x[2i] * cos(angle) - x[2i+1] * sin(angle)
    x'[2i+1] = x[2i] * sin(angle) + x[2i+1] * cos(angle)

Backward:
    Simply rotate by negative angle (transpose of rotation matrix)
    

RoPE Layouts: Pairwise vs Split-Half

One subtle but critical detail: RoPE is not only about the angle schedule. You also need the correct channel pairing layout. Two models can use the same rope_theta, the same cache, and the same attention code, yet still diverge badly if they pair channels differently.

(0,1), (2,3), (4,5), ...

x'[2i]   = x[2i] * cos - x[2i+1] * sin
x'[2i+1] = x[2i] * sin + x[2i+1] * cos

(0, d/2), (1, d/2+1), (2, d/2+2), ...

Pairwise:
  x[0] with x[1]
  x[2] with x[3]

Split-half:
  x[0] with x[d/2]
  x[1] with x[d/2+1]

GLM-4 RoPE: Pairwise Partial Rotary

Pairwise partial RoPE:
  rotate:      (x0, x1), (x2, x3), ... within rotary_dim
  pass-through: dimensions rotary_dim .. head_dim-1

GLM template/model map:
  rope_layout       = pairwise
  partial_rotary    = true
  rope_qk override  = rope_forward_qk_pairwise

Gemma4 RoPE: Per-Layer Direct Split-Half

template/gemma4.json
  attention_contract.rope_param_mode = per_layer_direct
  layer_rope_kind[]                  = full | sliding
  layer_rotary_dim[]                 = per-layer rotary width
  rope_theta / rope_theta_swa        = per-kind frequency base

lowered IR
  rope_qk = rope_forward_qk_split_direct
  attrs carry the layer rotary contract

kernel map
  rope_forward_qk_split_direct -> rope_forward_qk_split_direct_f32

Attention Mechanisms

Attention is the core of transformers: it lets each token "look at" all other tokens and decide what's relevant.

1. Project input to Q, K, V:
   Q = input @ W_q    # What am I looking for?
   K = input @ W_k    # What do I contain?
   V = input @ W_v    # What do I offer?

2. Compute attention scores:
   scores = Q @ K.T / sqrt(d_k)    # How relevant is each position?

3. Apply causal mask (decoder only):
   scores[i][j] = -inf if j > i    # Can't look at future tokens

4. Softmax to get weights:
   weights = softmax(scores)       # Normalize to probabilities

5. Weighted sum of values:
   output = weights @ V            # Aggregate information
    

d_k = 64:  scale = 1/8 = 0.125
d_k = 128: scale = 1/11.3 ≈ 0.088
    

Flash Attention

Flash Attention is an algorithmic optimization that computes exact attention without materializing the full N×N attention matrix. It's about memory efficiency, not approximation.

scores = Q @ K.T  # [N, N] matrix!

N = 2048:  16 MB
N = 8192:  256 MB
N = 32768: 4 GB
N = 131072: 64 GB  # Won't fit in GPU!

Traditional: softmax(x) = exp(x) / sum(exp(x))  # Need all x first

Online (Flash):
    For each new block of scores:
        1. Update running max: m_new = max(m_old, max(block))
        2. Rescale previous sum: sum *= exp(m_old - m_new)
        3. Add new block contribution: sum += sum(exp(block - m_new))
        4. Update running output with correction factor
    

Recurrent Attention and Gated DeltaNet

Some hybrid architectures avoid scanning a full KV-cache on every layer by carrying a compact recurrent state. In Qwen3.5/qwen3next-style layers, CK implements the Gated DeltaNet update as a dedicated kernel family rather than treating it as ordinary attention or a wrapper around GEMM.

Sliding-Window Attention

Sliding-window attention limits each query to the most recent W tokens. It is exact within the window, keeps compute linear in sequence length, and provides predictable latency for long contexts.

For token i:
  attend j in [max(0, i - W + 1), i]

scores[i][j] = -inf if j > i
scores[i][j] = -inf if i - j >= W
        

Grouped Query Attention (GQA)

GQA reduces memory and compute by sharing K, V heads across multiple Q heads.

Sparse / Query-Selected Attention (QSA)

Qwen3.8 Flash Next (PR #456) introduces a second way to avoid full-prefix attention: instead of restricting each query to a fixed recent window, a cheap indexing pass scores blocks of the prefix and attention runs over only the selected token indices. This is content-based selection, complementary to sliding windows.

Full causal attention at position t:   O(t)   score+softmax+V reads
QSA at position t:                     O(token_budget)  (+ cheap index scoring)

KV bytes read per query drop from the full prefix
to token_budget gathered entries — decode stays fast
long after a full KV cache would be bandwidth-bound.
    

Normalization: RMSNorm vs LayerNorm

Normalization stabilizes training by keeping activations in a reasonable range.

mean = sum(x) / n
var = sum((x - mean)²) / n
y = gamma * (x - mean) / sqrt(var + eps) + beta
        

rms = sqrt(sum(x²) / n)
y = gamma * x / (rms + eps)
        

Nemotron-H Hybrid Circuit: Attention + Mamba2 + MoE

Nemotron-H is important for CK because it is not just another dense transformer stack. A single model family can mix attention layers, Mamba2-style state-space layers, routed MoE blocks, ReLU2 experts, tokenizer guardrails, and model-specific template rules. CK needs to represent that as an auditable circuit of named kernels before it becomes a high-performance runtime.

Mamba2 Reference Kernels

Hybrid models such as Nemotron-H include Mamba2-style recurrent layers alongside attention and MoE blocks. These layers are not attention: they maintain a compact state, update it token by token, and use learned decay/input/output terms to produce the next hidden representation.

projected -> split(gate, hidden_bc, dt)
conv_state = roll_append(conv_state, hidden_bc)
dt = softplus(dt + dt_bias)
state = state * exp(dt * A) + dt * B * x
y = C dot state + D * x
out = grouped_rmsnorm(y * silu(gate))
        

Mixture of Experts: Router + ReLU2 Experts

MoE layers replace one dense MLP with a pool of experts and a router. For each token row, the router scores experts, selects a small top_k subset, runs only those expert MLPs, and accumulates the weighted result. This is sparse control flow wrapped around dense per-expert math.

scores = sigmoid(router_logits + correction_bias)
groups = partition(experts, n_group)
active_groups = topk(group_scores, topk_group)
experts = topk(scores inside active_groups, top_k)
weights = normalize(scores[experts])
        

for selected expert e:
  h1 = hidden @ W_up[e].T
  h2 = relu2(h1)
  y  = h2 @ W_down[e].T
  output += routing_weight[e] * y
        

Activations: ReLU2, SwiGLU, GeGLU, GELU

Activation functions introduce non-linearity. Modern LLMs often use gated activations (SwiGLU, GeGLU), while some Nemotron-family MLP blocks use simpler squared activations such as ReLU2. CK treats these as first-class kernels because each activation has a different forward/backward contract.

ReLU(x)  = max(0, x)
ReLU2(x) = max(0, x)^2

d ReLU2 / dx = 2x if x > 0 else 0
        

GELU(x) = x * Φ(x)
        ≈ x * sigmoid(1.702 * x)
        

Swish(x) = x * sigmoid(x)

SwiGLU(x) = Swish(x @ W_gate) * (x @ W_up)
        

Split input: x = [a | b]

GeGLU(x) = GELU(a) * b
GELU(a) = 0.5 * a * (1 + tanh(sqrt(2/pi) * (a + 0.044715 * a^3)))
    

gate = activation(x @ W_gate)  # Swish (SwiGLU) or GELU (GeGLU)
value = x @ W_up            # The actual content

output = gate * value       # Gate controls information flow
    

Vision Patch Embedding

Multimodal families (Qwen3-VL, Gemma4 Vision, the standalone SigLIP ViT tower) start by turning a raw image into a sequence of encoder tokens. There is no attention or MLP here yet — this is the frontend that the transformer body consumes. In CK-Engine these steps are concrete kernels in src/kernels/vision_kernels.c, not hidden preprocessing, which is what lets the vision encoder share the same lowering and memory plan as the text decoder.

im2patch:        [C,H,W] -> [N, C*P*P]    (N = (H/P)*(W/P))
patch embed:     [N, C*P*P] -> [N, d]     (dual proj + stream merge + bias)
spatial_merge_2x2: group 4 patches -> [N/4, 4*d]
+ tiled 2D pos:  position_embeddings_add_tiled_2d + 2D RoPE
        

How a Patch Knows Where It Is: 1D Text vs 2D Vision RoPE

Your intuition is exactly right. In text, a token's "position" is a single number — its index t in the sequence (0, 1, 2, …). RoPE turns that index into a rotation angle and rotates every (xi, xi+d/2) dimension pair of Q and K by it. Because rotation is relative, attention automatically recovers "how many tokens apart" two positions are. Position lives on one line.

An image patch has no single index that means anything — what matters is where on the grid it sits. So instead of one scalar, a patch token carries two coordinates: its row y and its column x on the patch grid (the grid being (H/P) × (W/P)). This is the "context is basically the row and the height/column of the i-th patch" idea you described.

CK encodes this two ways at once, both grounded in src/kernels/vision_kernels.c:

text   :  angle = t * base^(-2i/d)          // one index
vision :  y-dims rotate by  y * base^(-2i/d)
          x-dims rotate by  x * base^(-2i/d) // two indices
        

Audio Frontend: WAV to Log-Mel to Conv Stem

Audio families (Whisper Tiny today) start by turning a raw waveform into a sequence of encoder tokens — the same role the patch frontend plays for vision. In CK-Engine these steps are concrete kernels in src/kernels/audio_kernels.c (declared in include/ckernel_audio.h), not hidden preprocessing, which is what lets the audio encoder share the same lowering, memory plan, and parity-gate machinery as the text and vision lanes.

wav decode:  PCM16 -> f32, s=pcm/32768  [n_samples]
resample:    windowed sinc -> 16 kHz mono
pad/trunc:   extent -> 480,000 samples  (30 s)
stft power:  n_fft=400, hop=160, Hann   [201 x 3000]
mel filters: Slaney 80 x 201, built once
log-mel:     log10, clamp max-8, /4     [80 x 3000]
conv stem:   k3s1 + k3s2 + GELU         [384 x 1500]
transpose:   channel -> token major     [1500 x 384]
        

Kernel by Kernel: What Each Stage Actually Computes

Each box in the frontend diagram is a named kernel with a hand-checkable contract. The eight subsections below walk the math exactly as src/kernels/audio_kernels.c implements it — the same deep-dive treatment the other pages give attention and MLP. The full per-kernel deep dive, with inline SVGs and parity-gate evidence, lives on the Audio Kernels Deep Dive page.

1. WAV Decode: RIFF Chunks to Normalized Mono

audio_wav_decode_memory_pcm16_mono_f32 walks the RIFF container by hand: chunk sizes are odd-padded (+1), only the first fmt and data chunks are honored, and anything that is not PCM tag 1 / 16-bit is a typed rejection — there is no silent coercion. The sample conversion is channel-averaged, little-endian, scaled into [−1, 1):

mono[f] = (Σc int16_le( pcm[(f·C + c)·2] ) / C) · (1/32768)

For already-decoded interleaved PCM, audio_pcm_s16_to_mono_f32 applies the same math without the container walk. Deep dive →

2. Resample: Endpoint-Preserving Length, Two Interpolants

The output length is computed to preserve both endpoints — so a 30.00 s clip maps to exactly 480,000 samples at 16 kHz regardless of input rate:

n_out = 1 + (n_in − 1) · r_out / r_in
linear:  y[i] = fmaf(x[r+1] − x[l], frac, x[l])   (integer position math)
sinc:    w(d) = cutoff · sinc(cutoff·d) · hann(d/radius)
         y[i] = Σ w·x / Σ w                    (double position math)

The windowed-sinc path uses cutoff = min(1, r_out / r_in) as its anti-alias guard and clamps radius to [2, 128]; it is the band-limited contract path, while linear interpolation is the declared fast path. Deep dive →

3. Pad / Truncate: The Fixed 30-Second Extent

Everything downstream assumes a fixed time extent — the STFT frame count, the positional table, and the 1500-token cross-attention bridge all derive from exactly 480,000 samples. audio_pad_or_truncate_f32 makes that true for arbitrary input:

copied = min(n_in, 480000)
out[:copied] = in[:copied]   (memmove, overlap-safe)
out[copied:] = 0             (zero-pad short clips)

Short clips are zero-padded, long clips are truncated — not chunked — and the return value reports how many frames were real. Deep dive →

4. STFT Power: 20×20 Mixed-Radix FFT

Frames are center-padded (reflect) and windowed with a periodic Hann, then transformed by an FFT specialized to n_fft=400 — decomposed as 20×20 rather than padded to a power of two:

frame:   x[t·160 + n − 200]   reflect-padded (center = true)
window:  w[n] = 0.5 − 0.5·cos(2πn/400)
FFT:     400 = 20×20  (stage 1: DFT over x[p + 20q],
                        stage 2: combine with twiddle(f, p))
power:   P[t, f] = fmaf(re, re, im·im)   → [201 × T],  T = n/160

Trig tables and the window are built once by audio_stft_precompute_tables_f32; the power accumulation is FMA-fused so the magnitude-squared costs one pass. Deep dive →

5. Mel Filters: The Slaney Filterbank, Built Once

The 80×201 filterbank is constructed once, in double precision, by audio_whisper_mel_filters_slaney_f32 — evenly spaced on the Slaney mel scale, area-normalized so loudness sums comparably across bands:

mel points: n_mels + 2 evenly spaced, mel(0) .. mel(Nyquist)
w[m, bin] = max(0, min(lower, upper)) · 2/(right − left)

Promoting the filterbank to its own circuit op makes the Slaney identity a named, parity-gated artifact; log-Mel receives the matrix as data, the way PyTorch passes a buffer. Deep dive →

6. Log-Mel: Global 80 dB Clamp, Not Per-Frame

The mel projection is a plain matmul against precomputed Slaney-style filters, then the Whisper normalization — note the clamp is against the global max over the whole spectrogram, which is what keeps quiet passages quiet instead of normalizing every frame to full scale:

M  = mel_filters[80 × 201] @ power[201 × T]     (mel-major out)
L0 = log10(max(M, 1e-10))
L  = (max(L0, global_max − 8) + 4) / 4

Result: [80 × 3000] log-Mel features in roughly [−1.5, 0]. Deep dive →

7. Conv1D Stem: Two Strided Convolutions with GELU

The stem stays in the channel-major layout the weights arrive in. Conv1 lifts 80→384 channels (k3, s1, p1), conv2 keeps 384→384 but strides by 2 — that stride is what halves 3000 log-Mel frames into the 1500 encoder tokens:

y[o, t] = b[o] + Σi Σk w[o,i,k] · x[i, t·s + k − p]
conv1: 80 → 384, k3 s1 p1, GELU(erf)   [384 × 3000]
conv2: 384 → 384, k3 s2 p1, GELU(erf)  [384 × 1500]

Accumulation is FMA-fused and the padding is an explicit bounds check (zero-pad), so the kernel has no halo assumptions. Deep dive →

8. Transpose: The Single Named Layout Seam

Attention wants token-major rows; the conv stem produces channel-major. Rather than threading layout flags through every consumer, the frontend does one explicit transpose and names it:

out[t·C + c] = in[c·T + t]     [384 × 1500] → [1500 × 384]
row t = token t + learned 1D positional embedding

After this seam, the audio encoder is layout-identical to the text and vision lanes — the same attention and MLP kernels consume it without a Whisper-specific branch. Deep dive →

Cross-Attention: Decoder Queries, Encoder Memory

Everything above feeds a standard self-attention encoder. The genuinely new piece an encoder–decoder audio model adds is cross-attention: the decoder's queries read keys and values projected from the encoder's 1500 frames, not from its own sequence. That is the bridge every transcribed word is conditioned on — and it is exactly where the runbook's stage-split probe (encoder.npy handoff) cuts the system in two.

self-attn:   Q,K,V = xW_q, xW_k, xW_v   (same sequence)
cross-attn:  Q = xW_q                   (decoder, per token)
             K,V = H_enc W_k, H_enc W_v (encoder, once)

attn = softmax(Q K^T / sqrt(d_k)) V
     [1 x 384] x [384 x 1500] -> [1 x 1500] -> [1 x 384]
        

Weight Tying

Weight tying shares parameters between the input embedding and output projection, reducing model size with minimal quality loss.

Input embedding:  token_id → vector    E[vocab, hidden]
Output projection: vector → logits     W[vocab, hidden]

With weight tying: W = E  (same matrix!)
    

Hyper-Connections (Multi-Stream Residuals)

Qwen3.8 Flash Next (PR #456) replaces the single residual stream with S parallel hyper-streams: the hidden state is broadcast from [R, D] to [R, S*D] at layer entry, and each block writes back into selected streams instead of one shared residual. CK implements this as residual_topology: four_stream_gated_hyper_connection in the qwen4_exp circuit.

Per-Layer Embedding (PLE)

PLE gives every layer a cheap, token-context-sensitive input without running attention: a rolling n-gram window over recent token states is hashed into per-layer embedding tables, gated, smoothed by a causal conv, and injected into the hyper streams.

ring window: last n token states (EOS-padded)
hash n-gram -> per-layer embedding table
embed = table[hash] * layer_multipliers[layer]
        

group-RMSNorm(key/query) -> sigmoid gate
-> dilated causal depthwise conv (persistent conv_state)
-> inject into hyper streams
        

Summary: What Each Config Field Controls

Further Reading

• Kernel Reference - API documentation for each kernel
• Code Generation - How we generate C from configs
• Source Code - Browse the implementation

Diagram − + ×