Audio Kernels Deep Dive: From WAV to Encoder Tokens, Kernel by Kernel

The Whisper Tiny frontend turns a PCM16 WAV into the [1500 × 384] token matrix the audio encoder consumes. CK breaks that transformation into seven circuit operations across six frontend stages — STFT table construction and STFT execution are separate ops — each with one job, one layout contract, and one parity oracle. This page walks each kernel the same way the DeltaNet deep dive walks the recurrent update: the math, the way the C actually computes it, the memory layout, and the evidence that pins it to PyTorch.

Table of Contents

• 1. WAV Decode — audio_wav_decode_memory_pcm16_mono_f32
• 2. Resample — audio_resample_linear_f32 / audio_resample_windowed_sinc_f32
• 3. Pad / Truncate — audio_pad_or_truncate_f32
• 4. STFT Power — audio_stft_power_fft400_f32 (20×20 mixed radix)
• 5. Mel Filters — audio_whisper_mel_filters_slaney_f32
• 6. Log-Mel — audio_whisper_log_mel_from_power_reference_f32
• 7. Conv1D Stem — audio_conv1d_channel_major_f32
• 8. Token Transpose — audio_transpose_channel_to_token_f32
• 9. Supporting Kernels — position embeddings, erf GELU, head-major attention
• 10. The Encoder Body — four bidirectional blocks
• 11. Transformer-XL Relative Attention — position table, fused scores, relative shift
• 12. The FastConformer Convolution Module — GLU, depthwise Conv1D, inference BatchNorm
• 13. The LSTM Step — IFGO gates in PyTorch order
• 14. One Log-Mel Family, Three Contracts — Whisper vs Parakeet vs Cohere
• Parity & Nightly Coverage

1. WAV Decode: Bytes With a Contract, Not a Library Call

Most pipelines hand WAV parsing to a media library. CK walks the RIFF container by hand because the frontend's input contract is part of the numerical evidence: PCM (format tag 1), 16-bit, any channel count, any sample rate — anything else is rejected with a distinct error code before a single float is produced. audio_wav_parse_memory validates the container, audio_wav_decode_pcm16_mono_f32 turns it into mono f32, and the frontend circuit's map-owned provider audio_wav_decode_memory_pcm16_mono_f32 fuses the two into the single call the generated code makes. For audio that is already interleaved int16 in memory, audio_pcm_s16_to_mono_f32 applies the same conversion without the container walk.

2. Resample: Two Explicit Paths to 16 kHz

Whisper expects 16 kHz mono. CK exposes two named resamplers and the contract records which one a run used: audio_resample_linear_f32 for the fast path, and audio_resample_windowed_sinc_f32 for the band-limited path. Both share one frame-count convention — audio_resampled_frame_count returns 1 + (n_in − 1) · r_out / r_in, preserving both endpoints of the signal, so a file never grows or shrinks silently by a rounding choice.

3. Pad / Truncate: The Fixed 480,000-Sample Contract

Everything downstream of resampling assumes a fixed time extent: the STFT frame count, the encoder's positional embedding table, and the 1500-token cross-attention bridge all derive from exactly 480,000 samples (30 s at 16 kHz). audio_pad_or_truncate_f32 is the kernel that makes that true for arbitrary input — short clips are zero-padded, long clips are truncated (not chunked), and the function returns how many frames were real.

4. STFT Power: Whisper Geometry as a 20×20 Mixed-Radix FFT

The spectrum is where the frontend spends nearly all of its FLOPs, and where parity is easiest to break. Whisper's geometry is fixed: n_fft = 400, hop = 160, 201 bins, a periodic Hann window, and center = true with reflect padding. CK has three implementations of the same contract — a table-driven direct DFT (audio_stft_power_precomputed_f32), a Whisper-specialized mixed-radix FFT (audio_stft_power_fft400_f32), and the fully self-contained reference (audio_whisper_stft_power_reference_f32) that the nightly gate runs against PyTorch.

5. Mel Filters: The Slaney Filterbank, Built Once

Between the STFT power grid and the log-Mel projection sits the filterbank itself: 80 triangular filters over the 201 power bins, spaced evenly on the Slaney mel scale between 0 Hz and Nyquist. audio_whisper_mel_filters_slaney_f32 constructs the matrix once at table-build time — in double precision — so the per-frame work stays a single f32 matmul and the triangle geometry can never drift between runs.

6. Log-Mel: Perceptual Compression With a Global Clamp

audio_whisper_log_mel_from_power_reference_f32 turns the [201 × T] power spectrum into the [80 × T] log-Mel matrix the conv stem consumes. Three distinct numerical decisions live here, and all three match whisper.log_mel_spectrogram exactly: a Slaney-style triangular mel filterbank, a floored log10, and a global dynamic-range clamp before the final affine rescale. The fused entry point audio_whisper_log_mel_reference_f32 is just STFT + this kernel composed — the nightly gate checks the composition as well as the stages.

7. Conv1D Stem: The Only Learned Layers Before Attention

audio_conv1d_channel_major_f32 is the frontend's first weighted kernel — everything before it is fixed signal processing. The Whisper stem is two 3-tap convolutions with GELU between them: 80 → 384 at stride 1, then 384 → 384 at stride 2, which halves the time axis from 3000 log-Mel frames to the 1500 audio tokens the encoder (and the decoder's cross-attention) is built around. The kernel is deliberately generic — channels, kernel size, stride, and padding are parameters — because the same contraction serves any 1D conv frontend.

8. Token Transpose: The Named Layout Seam

audio_transpose_channel_to_token_f32 is eight lines of C and one of the most important kernels in the frontend. Everything upstream is channel-major ([C × T]) because convolutions want contiguous channels; everything downstream is token-major ([T × C]) because attention wants one row per sequence element. This kernel is the single explicit point where the layout flips — after it, the learned positional embedding is added and the matrix is literally the 1500-token sequence the encoder reads.

9. Supporting Kernels: The Shared Glue

Three library kernels complete the audio path without being Whisper-specific. position_embeddings_add applies the learned 1D table after the token transpose — the same kernel the vision lane uses for its 2D table. gelu_pytorch_erf_f32_inplace is the exact erf GELU the stem contracts pin (note the naming trap: gelu_exact_inplace is historically the tanh approximation despite its name). And attention_forward_query_key_head_major_f32 serves both attention oracles — the batched causal self-attention transition and the Q=1, K=1500 cross-attention decode step — with head-major rows that keep the cached K/V slices contiguous.

10. The Encoder Body: Four Bidirectional Blocks

Everything so far produced tokens. The encoder's job is to contextualize them: four identical pre-LayerNorm blocks turn the raw [1500 × 384] frame sequence into a memory where each frame's vector has absorbed meaning from the whole 30 seconds. The block contract is explicit in audio_transformer_encoder.json: layernorm → q/k/v_proj → attn → out_proj → residual → layernorm → mlp_up → gelu → mlp_down → residual, then one final LayerNorm.

The attention itself is where the audio encoder differs from the decoder you already know. Three facts, all declared in the circuit's attention_contract: it is dense bidirectional (causal: false) — the score grid is the full 1500×1500 square, not the decoder's causal triangle, so the frame under the word "country" borrows from frames before and after it. There is no RoPE — position comes from the absolute learned table added back at the transpose. And the K/V are ephemeral (ephemeral_full_context) — computed per layer, used once, freed; caching belongs to the decoder, not here. Per head (6 heads × 64 dims) the math is the usual softmax(Q·Kᵗ/√64)·V — softmax row-normalizes the square, so each frame's output is a weighted mixture of all 1500 value rows.

11. Transformer-XL Relative Attention: One Position Table, Two Score Kernels

The FastConformer encoders in Parakeet and Cohere do not use learned absolute positions or RoPE. Position enters only through the attention scores, Transformer-XL style: a sinusoidal table over relative offsets, two learned per-head bias vectors bias_u/bias_v, and a score that is the sum of a content term and a position term. CKE implements this as three providers. audio_relative_sinusoidal_position_f32 builds the [2T-1, E] table once per forward (descending offsets T-1 .. -(T-1), sin/cos interleaved). Each layer projects it with the existing GEMM provider into a head-interleaved relative_key. Then either the scores are computed fused — Parakeet's audio_conformer_relative_attention_f32 — or a full raw position-score matrix is materialized and re-indexed by audio_relative_shift_f32, which is the composition Cohere's encoder uses.

12. The FastConformer Convolution Module: GLU, Depthwise Conv1D, Inference BatchNorm

Every FastConformer block carries a convolution module between the attention and the second Macaron FFN: pointwise Conv1D (1024→2048) → GLU → depthwise Conv1D (K=9) → BatchNorm → SiLU → pointwise Conv1D (1024→1024), wrapped in a residual add. Three of its stages are the new providers — audio_conv1d_channel_major_grouped_f32 (one kernel covering pointwise, grouped, and depthwise via the groups argument, with an optional bias), audio_glu_split_channel_major_f32, and audio_batch_norm_inference_channel_major_f32. All of them operate on the same channel-major [C, T] layout: one contiguous 93-float row per channel on the worked fixture (width 1024, 93 frames after subsampling).

13. The LSTM Step: IFGO Gates in PyTorch Order, One Token at a Time

Parakeet's prediction network is a two-layer, 640-wide LSTM — the language-model half of the TDT transducer. audio_lstm_step_f32 advances it by exactly one token: one gate preactivation over the fused [4H, I]/[4H, H] weight matrices, one IFGO nonlinearity pass in PyTorch's gate order, and an in-place state update. The kernel is deliberately a step: the next input token depends on an argmax over the joint head, which depends on this LSTM's output, so there is no sequence to batch. State (h, c) is session-owned and persists across decode steps; the 4H gate buffer is caller-owned scratch shared by both layers; the kernel never touches the heap.

14. One Log-Mel Family, Three Contracts: Whisper vs Parakeet vs Cohere

Sections 4–6 walked the Whisper log-mel. Parakeet and Cohere reuse the same Conformer frontend provider family — pre-emphasis, centered STFT, Slaney mel, natural-log projection, per-feature normalization — but the contract differs at five points, and each difference changes the numbers the encoder was trained to expect. The table below is the exact divergence map, grounded in the two circuit contracts (parakeet_tdt.json, cohere_transcribe.json) and the two session runners that select the provider arguments.

Parity & Nightly Coverage

Diagram − + ×