Gated DeltaNet: Recurrent Attention Deep Dive

Per head (h = 0 … num_heads-1):

  q̂   = L2_norm(q) / √state_dim      ← scaled unit query
  k̂   = L2_norm(k)                    ← unit key
  β_s = σ(β)        = 1/(1+e^(-β))    ← write gate  (0,1)
  gate = exp(g)                        ← decay gate  (0,∞)

  S_decay = gate · S_prev              ← element-wise forget
  kv_mem  = S_decayᵀ · k̂              ← what memory recalls for k̂
  δ       = β_s · (v − kv_mem)         ← correction: new info minus old
  S_new   = S_decay + outer(k̂, δ)     ← rank-1 write to memory
  out     = S_newᵀ  · q̂               ← read from updated memory

═══════════════════════════════════════════════════════════════
GATED DELTANET (per head, per token)
═══════════════════════════════════════════════════════════════
State:    S ∈ ℝ^{d × d}     (matrix — stores key-value associations)
Decay:    S ← exp(g) · S     (element-wise scalar decay)
Retrieve: kv_mem = Sᵀ · k̂    (what does memory say for this key?)
Error:    δ = σ(β) · (v − kv_mem)  (correction signal)
Update:   S ← S + outer(k̂, δ)     (rank-1 write)
Output:   out = Sᵀ · q̂             (read with query)

Total per step:  O(d²)  — dominated by state matrix operations

═══════════════════════════════════════════════════════════════
MAMBA-2 SSM (per channel, per token)
═══════════════════════════════════════════════════════════════
State:    h ∈ ℝ^d            (vector — linear dynamical system)
Decay:    h ← A_t · h        (input-selective diagonal decay)
Inject:   h ← h + B_t · x_t  (input-selective injection)
Output:   y_t = C_t · h       (input-selective readout)

Conv:     Pre-SSM depthwise conv1d with learned kernel

Total per step:  O(d) for the recurrence + O(k·d) for conv1d

═══════════════════════════════════════════════════════════════
KEY DISTINCTION
═══════════════════════════════════════════════════════════════
DeltaNet writes CORRECTIONS. It asks "what is wrong with my memory?"
SSM writes INPUTS. It asks "what should I add to my state?"

DeltaNet has d² state. SSM has d state.
DeltaNet has one update rule. SSM has conv1d + recurrence.

Qwen3.5 strategy:
  Layer 0: DeltaNet (recurrent)
  Layer 1: DeltaNet (recurrent)
  Layer 2: DeltaNet (recurrent)
  Layer 3: Full Attention        ← exact recall every 4th layer
  Layer 4: DeltaNet (recurrent)
  ...

Nemotron 3 strategy:
  Layer 0: Mamba-2 SSM
  Layer 1: Mamba-2 SSM
  Layer 2: Mamba-2 SSM
  Layer 3: Full Attention        ← similar interleaving
  Layer 4: Mamba-2 SSM
  ...

Both keep ~75% of layers as constant-cost recurrent.
The remaining ~25% attention layers handle precise retrieval.

// ssm_kernels.c — forward pass structure

void ssm_conv1d_forward_ref(
    const float *conv_x,     // [num_seqs, num_channels, kernel_size-1 + num_tokens]
    const float *kernel,     // [num_channels, kernel_size]
    float *out,              // [num_seqs, num_tokens, num_channels]
    int kernel_size,         // typically 4
    int num_channels,        // model dim / num_heads
    int num_tokens,          // sequence length
    int num_seqs             // batch
)

// For each (seq, token, channel):
//   out[seq][tok][ch] = dot(conv_x[seq][ch][tok : tok+kernel_size], kernel[ch][:])
//
// This is a depthwise (channel-independent) causal convolution.
// Each channel has its own learned kernel of size k (typically 4).
// The conv_x buffer carries history from previous chunks (kernel_size-1 slots).

// recurrent_state_kernels.c — state pipeline

void recurrent_conv_state_update_forward(
    const float *state_in,   // previous history [num_seqs × channels × history_len]
    const float *q, *k, *v,  // new token projections
    float *conv_x,           // assembled conv buffer [num_seqs × channels × total_len]
    float *state_out,        // updated history for next call
    int history_len,         // kernel_size - 1
    int num_seqs, num_tokens,
    int q_dim, k_dim, v_dim
)

// Step 1: Copy old history into conv buffer positions [0 : history_len]
// Step 2: Append new tokens into positions [history_len : history_len + num_tokens]
// Step 3: Extract updated history (last history_len positions) → state_out
//
// This is the "shift register" that feeds the causal conv1d kernel.