Parakeet TDT Generated-C Transcription

CKE generates the Parakeet frontend, complete encoder, and TDT decoder as native model code. A C transcription host invokes the generated model and decodes its token output without a Python runtime. Earlier Python-orchestrated native-kernel measurements remain reference evidence, not measurements of this generated path. Diarization remains a separate milestone.

Status: implemented, tested, planned

How audio becomes text

The pipeline has four compute stages plus a decode state machine declared in the Parakeet circuit. The generated libmodel.so owns their execution; the native C host supplies WAV windows and consumes token output. run_parakeet_native_v8.py remains a separate Python-orchestrated reference for diagnosis. The frontend turns PCM samples into 128-bin log-mel frames; three stride-2 grouped Conv2D stages subsample 743 live frames to a 93×1024 token grid; 24 FastConformer blocks and a 1024→640 projection produce the 93×640 encoder memory; the transducer loop then walks that memory frame by frame.

Stage-to-artifact map

Every stage binds to declared artifacts: the circuit block, the selected providers, the kernel-map contracts, and the documentation that explains that layer of the system. Kernel maps are the resolver-facing contracts described in Kernel Maps — anatomy; provider math for the shared frontend family is walked stage by stage in the Audio Kernels Deep Dive.

The frontend, complete encoder schedule, and TDT decoder lower through IR into generated C in libmodel.so. The native host invokes ck_model_transcribe_audio_wav and native token decoding. The earlier ctypes session is retained as a diagnostic reference; its results do not certify generated-path performance. Codegen and X-Ray apply to the generated components.

audio_fastconformer_block_f32 is an explicit composite provider. Its declared subgraph is two half-scaled feed-forward branches, relative attention, the convolution module, residuals, and final LayerNorm. It fixes the current FP32 internal providers and rounding boundaries, requires caller-owned scratch, permits declared input/output aliasing, and exposes the block boundary rather than every fused internal tensor. No backward implementation is claimed; an unfused diagnostic or future backward path must implement the same declared subgraph contract.

TDT decode: blank handling and time advancement

The greedy TDT loop holds an LSTM hidden/cell state, a current token (initially blank 8192), a reused decoder projection cache, and a frame index into the 93-row encoder memory. The generated path owns this loop; ParakeetSession.decode is its historical reference. Each iteration:

1. Prediction network, conditionally. The two-layer LSTM is stepped only when the previously emitted token was nonblank (or on the first iteration). After a blank, the cached decoder projection is reused unchanged — blank frames do not advance the language-model state.
2. Joint. The current encoder row and the decoder projection are summed, passed through ReLU, and projected to 8,198 logits: 8,193 token logits (vocabulary 0–8191 plus blank 8192) followed by 5 duration logits.
3. Two independent argmaxes. The token is the argmax of the token logits; the duration is durations[argmax(duration_logits)] with the checkpoint's table [0, 1, 2, 3, 4].
4. Advance rule. If the token is blank and the predicted duration is 0, the duration is forced to 1 — a blank may never freeze the loop. A nonblank token with duration 0 is legal and holds the frame (multiple symbols on one encoder frame, bounded by max_symbols_per_step = 10). The frame index then advances by the duration.
5. Termination. The loop stops when the frame index reaches 93 (the encoder length). A hard bound of 10 × 93 = 930 steps raises TDT decoder exceeded its encoder-derived step bound instead of looping forever.

Worked example from the recorded native trajectory (see the diagram strip): the first two steps are blanks — step 1 predicts duration 0, forced to 1; step 2 predicts duration 4, skipping frames 1–4 (leading silence). Step 3 emits token 499 at frame 5, and only then does the LSTM step. Across the clip, 6 of the 50 recorded entries are blank (including the initial state), 44 are emissions, and the durations sum to exactly 93 frames × 0.08 s = 7.44 s of coverage for the 7.435 s clip. Token timestamps are emitted as cumulative duration × 0.08 s with the reference processor's punctuation-collapsing refinement (decode_token_timestamps); the short fixture matches all 44 reference timestamp entries exactly.

Runbook: reproduce the certified path

Choose the execution path. The current cks-v8-run audio convenience command dispatches Parakeet to the Python-orchestrated reference long-audio runner. It is useful for conversion and diagnostics, but is not the standalone generated-C path. For that path, build the generated model bundle and invoke the native host below. The historical runbook and its measurements are retained for reference.

Supported checkpoint. Exactly one: nvidia/parakeet-tdt-0.6b-v3 revision 541d1f99...defd82b (CC BY 4.0), converted with --arch parakeet_tdt --dtype fp32. Other revisions or dtypes are outside the certified claim.

Input requirements. Mono, 16 kHz, 16-bit PCM WAV (RIFF). Anything else is rejected with Parakeet frontend currently requires mono 16-bit PCM WAV at 16000 Hz; normalize first, e.g. ffmpeg -i in.mp3 -ac 1 -ar 16000 -c:a pcm_s16le out.wav. The committed fixture version/v8/test_assets/parakeet_tdt_0_6b_v3/2086-149220-0033.wav is the certified input.

Step 1 — download the pinned checkpoint (cold, once, ~2.4 GB):

hf download nvidia/parakeet-tdt-0.6b-v3 \
  --revision 541d1f99c6b0c3cd0b11a95167540bb8edefd82b \
  --local-dir .ck_models/parakeet-tdt-0.6b-v3

Step 2 — convert to BUMPWGT5 (cold, once): writes weights.bump (2,508,517,458 bytes in the recorded run), config.json, and weights_manifest.json. Conversion time is one-time setup cost and is excluded from the warm transcription timings below.

python version/v8/scripts/convert_safetensors_to_bump_v8.py \
  --checkpoint .ck_models/parakeet-tdt-0.6b-v3 \
  --output .ck_build/parakeet-tdt-0.6b-v3/weights.bump \
  --config-out .ck_build/parakeet-tdt-0.6b-v3/config.json \
  --manifest-out .ck_build/parakeet-tdt-0.6b-v3/weights_manifest.json \
  --arch parakeet_tdt --dtype fp32

Generated-C native host. With a Parakeet runtime bundle already compiled from the circuit and BUMP weights (libmodel.so, weights.bump, weights_manifest.map, and its declared native dependencies), build and run the C host. The optional final arguments set window and overlap seconds. The host requires mono PCM16 WAV at 16 kHz and emits JSON.

make ck-audio-transcribe-v8
./build/ck-audio-transcribe-v8 \
  /path/to/parakeet-runtime/libmodel.so \
  /path/to/parakeet-runtime/weights.bump \
  /path/to/parakeet-runtime/weights_manifest.map \
  recording.wav 180 30

The standalone generated-path certifier is certify_parakeet_generated_tdt_v8.py; its tests include tests/test_v8_parakeet_generated_tdt.py. Artifact-specific output quality and long-audio speed require the corresponding generated-runtime report.

Historical Python reference, step 3 — build the native kernel libraries (cold, once per checkout):

make build/libckernel_engine.so libckernel_audio.so

Historical Python reference, step 4 — run transcription (warm):

python version/v8/scripts/run_parakeet_native_v8.py \
  --model .ck_build/parakeet-tdt-0.6b-v3 \
  --audio version/v8/test_assets/parakeet_tdt_0_6b_v3/2086-149220-0033.wav \
  --fixture docs/notes/artifacts/parakeet_tdt_0_6b_v3_2086-149220-0033_fp32.npz \
  --decode --output .ck_build/parakeet-tdt-0.6b-v3/native-e2e.json

--fixture and --decode turn the run into a certification: without them the runner still transcribes but has nothing to compare against. --stop-after-layer -1|0|23 bisects the encoder against fixture boundaries for debugging.

Output format. The report ("kind": "cke.parakeet.native_e2e") records runtime-library and weight SHA-256 provenance, per-stage seconds, frontend/encoder/first-logit RMSE comparisons, the full token-ID and duration sequences, the first divergent index for each, the detokenized transcript, punctuation-collapsed token timestamps, a checks map (including transcript_exact and timestamps_exact when a reference is supplied), and an overall "status": "pass". The recorded evidence is the committed native residual/argmax E2E report (44/44 timestamps exact).

Reference-path evidence and warm timing, not generated-C performance. On the recorded run (commit 909a2c164, PR #504, Ryzen 9 9950X3D per the README evidence table, FP32, workload = the 7.435-second fixture; the report JSON records platform and Python 3.14.4 but not thread count): frontend 0.069 s, encoder 1.027 s, total run 2.25 s including hashing, decode, and tokenizer work that sit outside the two stage timers. Expected output transcript: “Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait.”

When it fails: where to look

Frozen reference

The checkpoint metadata says 5.6.0.dev0. The released v5.6.0 source does not contain ParakeetForTDT, so certification pins the executable source commit directly.

First reference fixture

The 7.435-second mono 16 kHz utterance 2086-149220-0033.wav is the sample linked by NVIDIA's model card and comes from the CC BY 4.0 LibriSpeech corpus. On Ryzen with PyTorch 2.13.0+cpu, 16 threads, and the pinned Transformers commit, the reference completed in 0.39 seconds and produced 50 decode steps, 44 nonblank emissions, durations, and token timestamps.

Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait.

The compressed tensor fixture retains audio, normalized frontend features and mask, subsampling, encoder layers 0 and 23, projected encoder output, initial decoder output, all 8,198 first-step joint logits, generated token IDs, and durations. Every recorded array is finite and has an independent content hash.

• Fixture report (JSON)
• Intermediate tensors (NPZ)
• Complete 723-tensor manifest (JSON)
• Operation-reuse inventory (JSON)
• Remaining host-math ownership inventory (JSON)

Historical Python-reference execution evidence

The short native path converts all 699 inference tensors to BUMPWGT5 and excludes 24 BatchNorm training counters explicitly. On Ryzen, the frontend, all 24 FastConformer blocks, prediction network, joint network, greedy TDT decoder, tokenizer, and token timestamps complete without importing PyTorch or Transformers. The retained short result matches 50 of 50 token IDs, 50 of 50 durations, the decoded transcript, and all 44 timestamp entries.

The five-minute fixture consumes all 4,800,000 source samples and produces a complete transcript and monotonic token timestamps. Two runs produced identical token IDs, durations, transcript, and 1,296 timestamp entries. The retained repeat took 369.19 seconds (RTF 1.23) and 2.87 GB peak RSS on Ryzen. Its WER is 10.43% against the curated 690-word transcript.

Whisper Base on the identical PCM completed in 10.19 seconds (RTF 0.034) with 5.65% WER using its per-window fallback. Whisper's default persistent-worker path failed before inference with a dynamic-module pickling error, which is retained as a separate runtime limitation. Full Parakeet attention uses one caller-owned score row per head and schedules heads independently; the provider does not allocate heap memory. The complete selected call chain still needs repeated-call allocation instrumentation before an allocation-free runtime claim.

• Exact short native report (JSON)
• Exact short report after native residual and argmax integration (JSON)
• Five-minute native report (JSON)
• Five-minute repeat and quality certification (JSON)
• Matched Parakeet and Whisper result (JSON)
• Two-window short-audio reconciliation result (JSON)
• Five-minute chunking certification (JSON)
• 42-minute certification summary (JSON)

A synthetic 48 kHz stereo transform of the short fixture also reproduces the 16 kHz mono trajectory after CKE windowed-sinc resampling. This validates the wiring and deterministic fixture transform; broader real-world resampling quality is not yet certified.

The bounded long-audio policy uses 180-second full-attention windows with 30 seconds of overlap. Encoder and decoder state reset for each window. Each word belongs to the window whose ownership region contains its timestamp midpoint; ownership regions meet at the midpoint of each overlap. Two five-minute runs produced identical 700-word trajectories, exact 0–300 second coverage, and 7.25% WER. The retained repeat took 231.19 seconds (RTF 0.771) with 3.24 GB peak RSS.

The encoder's scaled and unscaled residual additions and both TDT argmax decisions now use declared native providers. On the clean Ryzen build at commit d3a6a8e55, this path preserved all 50 reference token IDs and all 50 duration decisions exactly. The machine-readable host-math inventory distinguishes the remaining model, layout, session, and planner ownership work.

Run the reference front door

The current Python audio front door recognizes the checkpoint architecture, creates the FP32 BUMP bundle, builds native kernel libraries, and selects the reference long-audio policy. It is not the standalone generated-C host shown above. The output report can resume only at completed window boundaries.

version/v8/scripts/cks-v8-run audio \
  hf://nvidia/parakeet-tdt-0.6b-v3 \
  --wav recording.wav \
  --output build/parakeet-transcript.json \
  --resume

Original operation-reuse inventory

This table records the pre-implementation inventory from PR #503: reusable meant the provider's interface fit, candidate reuse meant an interface fit still required comparison at the named fixture boundary, and missing composition meant the mathematical pieces existed without a declared Parakeet graph. PRs #504 and #507 closed the missing provider/runtime/composition rows with the providers named in the stage map and the parakeet_tdt circuit; the dispositions below are the historical record in the inventory JSON, whose top-level status is now native_five_minute_chunked_e2e_pass.

Implementation milestones

New production compute paths must use planner-declared scratch or session-owned state. They may not allocate or free heap memory per call, hide allocation in helpers, substitute large stack buffers, or share mutable scratch across sessions. The Parakeet providers follow this: attention uses one caller-owned score row per head; the LSTM uses caller-owned 4H gate scratch and session-owned state (guarded by tests/test_v8_parakeet_native.py).

Reproduce the reference capture

Download the exact Hugging Face revision once into the shared model cache, then run the fixture tool. Transformers, PyTorch, librosa, and soundfile belong to this certification environment; the CKE inference path does not depend on them.

export CKE_MODEL_CACHE="$HOME/.cache/cke/models"
hf download nvidia/parakeet-tdt-0.6b-v3 \
  --revision 541d1f99c6b0c3cd0b11a95167540bb8edefd82b \
  --local-dir "$CKE_MODEL_CACHE/parakeet-tdt-0.6b-v3-541d1f99"

PYTHONPATH="$HOME/.cache/cke/reference/transformers-66799f45" \
python version/v8/scripts/capture_parakeet_reference_v8.py \
  --model-dir "$CKE_MODEL_CACHE/parakeet-tdt-0.6b-v3-541d1f99" \
  --audio version/v8/test_assets/parakeet_tdt_0_6b_v3/2086-149220-0033.wav \
  --output-json /tmp/parakeet-reference.json \
  --output-npz /tmp/parakeet-reference.npz \
  --model-revision 541d1f99c6b0c3cd0b11a95167540bb8edefd82b \
  --transformers-revision 66799f45cea7513712580c6170cdaa4438df702a

Next work

The historical Python-reference run consumed the complete 42-minute recording with overlap coverage ending at 2,542.0 seconds. Its 7,225-word output has no aligned human transcript, so this result establishes execution, coverage, timestamp, and resource behavior for that path rather than WER or generated-path speed. The full-attention diagnostic runner fails closed above the checkpoint's 5,000-position limit. Generated-C execution is now implemented separately; multilingual coverage and diarization remain separate evidence lanes.

Diagram − + ×