Audio Kernels Deep Dive: From WAV to Encoder Tokens, Kernel by Kernel

The Whisper Tiny frontend turns a PCM16 WAV into the [1500 × 384] token matrix the audio encoder consumes. CK breaks that transformation into seven circuit operations across six frontend stages — STFT table construction and STFT execution are separate ops — each with one job, one layout contract, and one parity oracle. This page walks each kernel the same way the DeltaNet deep dive walks the recurrent update: the math, the way the C actually computes it, the memory layout, and the evidence that pins it to PyTorch.

Table of Contents

• 1. WAV Decode — audio_wav_decode_memory_pcm16_mono_f32
• 2. Resample — audio_resample_linear_f32 / audio_resample_windowed_sinc_f32
• 3. Pad / Truncate — audio_pad_or_truncate_f32
• 4. STFT Power — audio_stft_power_fft400_f32 (20×20 mixed radix)
• 5. Mel Filters — audio_whisper_mel_filters_slaney_f32
• 6. Log-Mel — audio_whisper_log_mel_from_power_reference_f32
• 7. Conv1D Stem — audio_conv1d_channel_major_f32
• 8. Token Transpose — audio_transpose_channel_to_token_f32
• 9. Supporting Kernels — position embeddings, erf GELU, head-major attention
• 10. The Encoder Body — four bidirectional blocks
• Parity & Nightly Coverage

1. WAV Decode: Bytes With a Contract, Not a Library Call

Most pipelines hand WAV parsing to a media library. CK walks the RIFF container by hand because the frontend's input contract is part of the numerical evidence: PCM (format tag 1), 16-bit, any channel count, any sample rate — anything else is rejected with a distinct error code before a single float is produced. audio_wav_parse_memory validates the container, audio_wav_decode_pcm16_mono_f32 turns it into mono f32, and the frontend circuit's map-owned provider audio_wav_decode_memory_pcm16_mono_f32 fuses the two into the single call the generated code makes. For audio that is already interleaved int16 in memory, audio_pcm_s16_to_mono_f32 applies the same conversion without the container walk.

2. Resample: Two Explicit Paths to 16 kHz

Whisper expects 16 kHz mono. CK exposes two named resamplers and the contract records which one a run used: audio_resample_linear_f32 for the fast path, and audio_resample_windowed_sinc_f32 for the band-limited path. Both share one frame-count convention — audio_resampled_frame_count returns 1 + (n_in − 1) · r_out / r_in, preserving both endpoints of the signal, so a file never grows or shrinks silently by a rounding choice.

3. Pad / Truncate: The Fixed 480,000-Sample Contract

Everything downstream of resampling assumes a fixed time extent: the STFT frame count, the encoder's positional embedding table, and the 1500-token cross-attention bridge all derive from exactly 480,000 samples (30 s at 16 kHz). audio_pad_or_truncate_f32 is the kernel that makes that true for arbitrary input — short clips are zero-padded, long clips are truncated (not chunked), and the function returns how many frames were real.

4. STFT Power: Whisper Geometry as a 20×20 Mixed-Radix FFT

The spectrum is where the frontend spends nearly all of its FLOPs, and where parity is easiest to break. Whisper's geometry is fixed: n_fft = 400, hop = 160, 201 bins, a periodic Hann window, and center = true with reflect padding. CK has three implementations of the same contract — a table-driven direct DFT (audio_stft_power_precomputed_f32), a Whisper-specialized mixed-radix FFT (audio_stft_power_fft400_f32), and the fully self-contained reference (audio_whisper_stft_power_reference_f32) that the nightly gate runs against PyTorch.

5. Mel Filters: The Slaney Filterbank, Built Once

Between the STFT power grid and the log-Mel projection sits the filterbank itself: 80 triangular filters over the 201 power bins, spaced evenly on the Slaney mel scale between 0 Hz and Nyquist. audio_whisper_mel_filters_slaney_f32 constructs the matrix once at table-build time — in double precision — so the per-frame work stays a single f32 matmul and the triangle geometry can never drift between runs.

6. Log-Mel: Perceptual Compression With a Global Clamp

audio_whisper_log_mel_from_power_reference_f32 turns the [201 × T] power spectrum into the [80 × T] log-Mel matrix the conv stem consumes. Three distinct numerical decisions live here, and all three match whisper.log_mel_spectrogram exactly: a Slaney-style triangular mel filterbank, a floored log10, and a global dynamic-range clamp before the final affine rescale. The fused entry point audio_whisper_log_mel_reference_f32 is just STFT + this kernel composed — the nightly gate checks the composition as well as the stages.

7. Conv1D Stem: The Only Learned Layers Before Attention

audio_conv1d_channel_major_f32 is the frontend's first weighted kernel — everything before it is fixed signal processing. The Whisper stem is two 3-tap convolutions with GELU between them: 80 → 384 at stride 1, then 384 → 384 at stride 2, which halves the time axis from 3000 log-Mel frames to the 1500 audio tokens the encoder (and the decoder's cross-attention) is built around. The kernel is deliberately generic — channels, kernel size, stride, and padding are parameters — because the same contraction serves any 1D conv frontend.

8. Token Transpose: The Named Layout Seam

audio_transpose_channel_to_token_f32 is eight lines of C and one of the most important kernels in the frontend. Everything upstream is channel-major ([C × T]) because convolutions want contiguous channels; everything downstream is token-major ([T × C]) because attention wants one row per sequence element. This kernel is the single explicit point where the layout flips — after it, the learned positional embedding is added and the matrix is literally the 1500-token sequence the encoder reads.

9. Supporting Kernels: The Shared Glue

Three library kernels complete the audio path without being Whisper-specific. position_embeddings_add applies the learned 1D table after the token transpose — the same kernel the vision lane uses for its 2D table. gelu_pytorch_erf_f32_inplace is the exact erf GELU the stem contracts pin (note the naming trap: gelu_exact_inplace is historically the tanh approximation despite its name). And attention_forward_query_key_head_major_f32 serves both attention oracles — the batched causal self-attention transition and the Q=1, K=1500 cross-attention decode step — with head-major rows that keep the cached K/V slices contiguous.

10. The Encoder Body: Four Bidirectional Blocks

Everything so far produced tokens. The encoder's job is to contextualize them: four identical pre-LayerNorm blocks turn the raw [1500 × 384] frame sequence into a memory where each frame's vector has absorbed meaning from the whole 30 seconds. The block contract is explicit in audio_transformer_encoder.json: layernorm → q/k/v_proj → attn → out_proj → residual → layernorm → mlp_up → gelu → mlp_down → residual, then one final LayerNorm.

The attention itself is where the audio encoder differs from the decoder you already know. Three facts, all declared in the circuit's attention_contract: it is dense bidirectional (causal: false) — the score grid is the full 1500×1500 square, not the decoder's causal triangle, so the frame under the word "country" borrows from frames before and after it. There is no RoPE — position comes from the absolute learned table added back at the transpose. And the K/V are ephemeral (ephemeral_full_context) — computed per layer, used once, freed; caching belongs to the decoder, not here. Per head (6 heads × 64 dims) the math is the usual softmax(Q·Kᵗ/√64)·V — softmax row-normalizes the square, so each frame's output is a weighted mixture of all 1500 value rows.

Parity & Nightly Coverage