Whisper Tiny and Base End-to-End

Kernel Source
src/kernels/audio_kernels.c and include/ckernel_audio.h — WAV decode, resampling, pad/truncate, STFT/FFT-400, Slaney Mel, log-Mel, Conv1D stem, and layout primitives. The frontend is the circuit version/v8/circuits/whisper_audio_frontend.json; encoder and decoder topology lives in version/v8/circuits/audio_transformer_encoder.json and version/v8/circuits/audio_transformer_decoder.json; lifecycle orchestration in version/v8/scripts/ck_run_v8.py (unified cks-v8-run audio command). Per-kernel math and diagrams: Concepts — Audio Frontend & Cross-Attention; per-kernel deep dive: Audio Kernels Deep Dive.

CKE v8 runs generated FP32 Whisper Tiny and Base encoders and decoders from a PCM16 WAV file — and the frontend is generated too. WAV decode, windowed-sinc resampling, pad/truncate, STFT table construction, FFT400 power, Slaney Mel filters, and log-Mel normalization are seven explicit circuit operations in version/v8/circuits/whisper_audio_frontend.json, each with an exact numerical contract and a map-owned call ABI. The resolved operation sequence lowers through call IR into generated C exporting ck_model_run_audio_wav. Python coordinates artifact lifecycles and process isolation; it performs no frontend arithmetic. Codegen contains no Whisper model-name branch; every arithmetic boundary resolves through a kernel-map contract.

How the Audio Pipeline Works

The diagram follows one WAV file through the full system: contract kernels prepare the log-Mel grid, the generated encoder writes an immutable encoder memory once, and the generated decoder reads it on every cross-attention step while its own KV-cache grows token by token.

Generated Frontend: Circuit Ops to Generated C

The frontend began as direct C calls selected by the Python runner — useful correctness bring-up, but provider selection sat outside the v8 architecture. It is now seven explicit circuit operations (audio_wav_decode, audio_resample, audio_pad_or_truncate, audio_stft_tables, audio_stft, audio_mel_filters, audio_log_mel), each carrying an exact numerical contract and a map-owned call ABI. Lowering plans the frontend buffers and lowers the runtime WAV metadata into call IR; the resolved sequence generates C exporting ck_model_run_audio_wav.

Whisper audio frontend lowering from circuit operations through kernel maps and call IR to generated C

Provider selection is a kernel-map decision. The map alone names the registered provider and its call ABI; tests fail closed when an operation is missing or the provider inventory bypasses map-owned ABI accounting.
Scheduling is a call-IR decision. Operation order and buffer lifetimes come from the circuit through lowering — not from Python control flow, and not from a codegen branch.
Python owns lifecycle, generated C owns arithmetic. The runner coordinates artifact directories, encoder/decoder process isolation, and report writing; every sample-level computation executes inside the generated artifact.

What Nightly Proves, and What Stays Opt-In

Coverage is split deliberately, because GitHub runners do not carry the generated Whisper artifacts:

Nightly — Audio Log-Mel Frontend. The optimized C FFT/log-Mel frontend is compared against a PyTorch oracle on every nightly run.
Nightly — Audio Transformer Primitives. Resampling, FFT, Conv1D, GELU, layout transforms, self-attention, and cross-attention are exercised against their contracts on every nightly run.
Nightly — v8 audio circuit/codegen regression. The generated-circuit lowering and codegen path is gated portably, without model weights. These rows now cover the full frontend sequence: native arithmetic, exact provider resolution for all seven ops, the generated ck_model_run_audio_wav ABI, unified-command CLI ownership, and synthetic compilation.
Opt-in artifact test — full model-and-WAV transcription. End-to-end transcription runs where the generated encoder and decoder artifacts exist, via make test-whisper-e2e-auto. The gate requires the full reference token sequence and EOS without tolerance relaxation.

Local YouTube And Meeting Transcription

CKE can be used as a local CPU transcription pipeline. The recording and generated transcript remain on the machine unless the operator separately uploads them. CKE does not bypass access controls or download private meeting recordings: export a recording you are authorized to process, then provide the local media file to the pipeline.

Install the media tools once. Distribution packages may also be used:

sudo apt install ffmpeg
python3 -m pip install --user yt-dlp

YouTube Audio

Download audio only from content you are permitted to access, then normalize it to the frontend's mono 16 kHz PCM16 WAV contract:

mkdir -p "$HOME/cke-audio/results"

yt-dlp -x --audio-format wav \
  -o "$HOME/cke-audio/source.%(ext)s" \
  "YOUTUBE_URL"

ffmpeg -y \
  -i "$HOME/cke-audio/source.wav" \
  -ac 1 -ar 16000 -c:a pcm_s16le \
  "$HOME/cke-audio/youtube-16k.wav"

Teams Or Other Meeting Exports

Download the recording through the meeting service's authorized export controls. MP4, M4A, MP3, FLAC, and other formats use the same normalization step:

ffmpeg -y \
  -i "$HOME/Downloads/meeting-recording.mp4" \
  -ac 1 -ar 16000 -c:a pcm_s16le \
  "$HOME/cke-audio/meeting-16k.wav"

Build And Run Whisper Base

The unified v8 command downloads the Hugging Face safetensors checkpoint, generates the encoder and decoder runtimes, compiles them, and transcribes the WAV. Subsequent runs reuse the validated generated artifacts:

RUN_DIR="$HOME/.cache/ck-engine-v8/models/whisper-base-local"

version/v8/scripts/cks-v8-run audio hf://openai/whisper-base \
  --run "$RUN_DIR" \
  --wav "$HOME/cke-audio/youtube-16k.wav" \
  --language en \
  --task transcribe \
  --max-tokens 448 \
  --output "$HOME/cke-audio/results/transcript.json"

To use already-built artifacts without checking or downloading the source checkpoint again:

CK_NUM_THREADS=20 OMP_NUM_THREADS=20 \
version/v8/scripts/cks-v8-run audio \
  --encoder-run-dir "$RUN_DIR/encoder" \
  --decoder-run-dir "$RUN_DIR/decoder" \
  --wav "$HOME/cke-audio/youtube-16k.wav" \
  --language en \
  --task transcribe \
  --max-tokens 448 \
  --output "$HOME/cke-audio/results/transcript.json"

The transcript is printed to standard output. The JSON report also preserves stage timings, generated tokens, audio identity, and runtime provenance. Extract only the stitched text with:

jq -r '.decoder.transcript_text' \
  "$HOME/cke-audio/results/transcript.json" \
  > "$HOME/cke-audio/results/transcript.txt"

Replace the WAV path with meeting-16k.wav for the meeting workflow. Add --timestamps when timestamp tokens are required. Plain transcription is the more mature long-recording path.

A local safetensors checkpoint directory can replace the hf:// argument. The command builds distinct encoder and decoder runtime bundles. It reuses them only when checkpoint, circuit, kernel registry, converter/codegen, generated model, and runtime hashes agree.

Encoder and decoder execution occurs in isolated child processes. Both generated artifacts export the same model ABI and shared-library symbol names, so loading both into one process can bind a call to the wrong artifact.

Current Limitations

Whisper Tiny, Base, and Small, FP32 only.
PCM16 WAV input; use ffmpeg to convert MP3, FLAC, or other formats.
Recordings longer than 30 seconds are processed as sequential source windows. A 33-second PCM16 fixture is parity-certified; arbitrary hour-scale recordings are functional but not yet corpus-certified.
English text-only and paired monotonic timestamp decoding are certified on the public JFK fixture.
The JFK sample matches Hugging Face exactly; arbitrary recordings are functional rather than corpus-certified yet.
CKE currently emits transcript JSON and plain text. It does not yet emit SRT or VTT subtitle files directly.

Correctness Findings

Cross-attention K/V projections must use the immutable encoder-memory length. Generic prefill code previously replaced their matrix row count with the active decoder-token count.
Cross-attention query length must use the active decoder-token count. The generated prefill call previously retained the decoder's physical capacity.
Incremental self-attention must use the decode provider and attend model->pos + 1 valid KV rows. A causal prefill provider invoked with one active token attended only cache row zero.
Encoder-side cross-attention K/V projections are immutable for an audio segment. The circuit declares a prefill-populated, layer-major FP32 cache; decode consumes the cached head-major slices instead of projecting and transposing all 1,500 encoder rows for every generated token. Binding new encoder memory invalidates the cache, and decode fails closed until prefill repopulates it.

These dimensions are now represented in call IR and phase-specific circuit contracts. Codegen consumes the resolved semantics and contains no Whisper model-name branch.

Reference Evidence

On the public JFK Whisper sample, CKE emitted the same 23 greedy text tokens as Hugging Face Whisper Tiny, followed by EOS:

And so my fellow Americans ask not what your country can do for you
ask what you can do for your country.

The optimized C FFT/log-Mel frontend differed from Hugging Face by approximately 1.54e-4 maximum absolute error and 1.37e-6 RMSE, without changing any generated token. Before persistent cross-attention caching, one measured run used 0.16 seconds for the frontend, 11.17 seconds for the encoder, 0.75 seconds for decoder prefill, and 16.57 seconds for 23 decode tokens. With the cache, the same exact token sequence used 0.12 seconds for the frontend, 11.15 seconds for the encoder, 0.73 seconds for prefill, and 0.21 seconds for decode. Decoder time improved by approximately 79x, and total time fell from 28.64 seconds to approximately 12.22 seconds.

Whisper Base emits the same 25 greedy text tokens and EOS as Hugging Face on the same sample, including both commas:

And so my fellow Americans, ask not what your country can do for you,
ask what you can do for your country.

At 20 threads, the measured Base run used approximately 24.43 seconds for the generated frontend plus encoder, 2.08 seconds for decoder prefill, and 0.43 seconds for 25 decode tokens. This certifies transcript parity on the fixture, not complete tensor parity. The synthetic encoder X-Ray first crosses its material threshold at layer 2 MLP-down and reaches roughly 9.77e-4 after final LayerNorm, so the transcript and tensor gates remain separate.

Same-Host Base Performance Baseline

A current three-repetition same-host AVX2 measurement compared the same 11-second JFK WAV, Whisper Base model family, English greedy transcript, 20 CPU threads, and 25 generated text tokens. Backend order rotated each repetition, and all three backends emitted identical text-token IDs:

Backend	Measured compute	Process wall time	CKE ratio vs native reference
CKE generated FP32	26.751 s	27.418 s	49.15×
PyTorch CPU	0.500 s	2.937 s	—
whisper.cpp FP16	0.544 s	0.567 s	1.00×

Compute and process wall time are intentionally separate. PyTorch compute includes feature extraction and generation but excludes checkpoint loading; its process wall time includes interpreter and framework startup. whisper.cpp total time includes its model load. CKE compute is the generated audio/encoder, decoder-prefill, and decode stage sum. This baseline proves that CKE's immediate audio priority is encoder scheduling and GEMM/provider optimization, not incremental decoder tuning.

python benchmarks/compare_whisper_backends_v8.py \
  --checkpoint /path/to/openai--whisper-base \
  --encoder-run-dir "$RUN_DIR/encoder" \
  --decoder-run-dir "$RUN_DIR/decoder" \
  --wav /path/to/jfk.wav \
  --whisper-cpp-cli /path/to/whisper.cpp/build/bin/whisper-cli \
  --whisper-cpp-model /path/to/whisper.cpp/models/ggml-base.bin \
  --language en --task transcribe \
  --threads 20 --repetitions 3 \
  --output build/reports/whisper-base-three-backend.json

The benchmark rotates backend order between repetitions, records commands, model and runtime hashes, CPU features, wall and compute timing, transcript text, and token IDs, and fails when any backend changes the transcript.

Timestamp mode also matches Hugging Face exactly: 27/27 generated tokens, including <|0.00|> and <|11.00|>, followed by EOS. The decoder contract owns the initial timestamp window, paired-token sequence, monotonic ordering, and aggregate timestamp-probability selection. Whisper Small reuses the same generated circuits and matched all 25 text tokens through EOS on the fixture; no model-size branch was added.

Regression Gates

make test-whisper-long-audio-nightly \
  V8_WHISPER_LONG_AUDIO_MODEL=base

The scheduled five-minute corpus runs the same published MIC1 recording through Tiny, Base, Small, Medium, and Large-v3. Each model must build and consume the full recording with monotonic bounded timestamps, forward window progress, sufficient transcript content, and its declared word-error ceiling. Missing models, missing reports, and unavailable artifacts fail the model job rather than becoming skips.

make test-audio

CK_WHISPER_ENCODER_RUN_DIR=/path/to/encoder \
CK_WHISPER_DECODER_RUN_DIR=/path/to/decoder \
CK_WHISPER_WAV=/path/to/jfk.wav \
make test-whisper-e2e-auto

CK_WHISPER_BASE_ENCODER_RUN_DIR=/path/to/base/encoder \
CK_WHISPER_BASE_DECODER_RUN_DIR=/path/to/base/decoder \
CK_WHISPER_BASE_WAV=/path/to/jfk.wav \
make test-whisper-e2e-auto

CK_WHISPER_CHECKPOINT=/path/to/openai--whisper-base \
CK_WHISPER_ENCODER_RUN_DIR=/path/to/base/encoder \
CK_WHISPER_DECODER_RUN_DIR=/path/to/base/decoder \
CK_WHISPER_WAV=/path/to/jfk.wav \
CK_WHISPER_TIMESTAMPS=1 \
make test-whisper-pytorch-e2e-auto

Nightly publishes separate PyTorch-backed audio frontend and transformer primitive rows, plus a portable v8 audio circuit/codegen row. Scheduled runs additionally execute the non-skippable five-minute model matrix. Together they fix the Slaney filter identity, forced decoder prefix, projection extents, phase-specific attention providers, generated call arguments, long-audio window progress, and timestamp bounds. The short JFK artifact gate remains available for exact token and EOS comparison.