Cohere Kernel Story: Reuse, BF16 Vision, and the Transcribe Audio Stack

Four Cohere bring-ups, four different relationships to the kernel registry: Command R mostly composed existing providers, North composed them entirely, North Micro Vision added a BF16 image frontend, and Transcribe added a seven-provider audio foundation whose encoder is still not promoted. This page walks the kernels each one touched — and is explicit about what is certified versus what is still a provider foundation.

1. Cohere2 Command R: the reuse story

Command R is structurally novel — a single shared pre-block LayerNorm feeds the attention and MLP branches in parallel, layers alternate sliding and full attention 3:1, the LM head is weight-tied, and per-tensor quant is mixed through the weight_dtype_registry (the weight policy simply ignores the duplicate ln2_* tensors the checkpoint carries). Yet the circuit version/v8/circuits/cohere2.json binds almost entirely to the existing Llama-family provider stack:

cohere2.json kernel bindings (all pre-existing except the last): layernorm = layernorm_fp32_exact rope_init = rope_precompute_cache_llama_cpu rope_qk = rope_forward_qk_pairwise (+ _llama_cpu decode variant) attn = attention_forward_causal_head_major_gqa_flash_strided attn_sliding = attention_forward_causal_head_major_gqa_flash_strided_sliding attn_decode = attention_forward_decode_head_major_gqa_flash[_sliding] final_logit_scale = final_logit_scale_f32 <-- the one new kernel

The single new provider is final_logit_scale_f32 (src/kernels/logit_kernels.c:7): an in-place FP32 logits[i] *= logit_scale footer op, driven by GGUF metadata rather than a hardcoded constant. Everything else was already hardened for other families.

click / tap the diagram to expand

The block-structure diagram lives in Architecture Variants — Cohere2; provider math in kernel architecture — norms, footer ops.

2. Cohere North (MoE): zero new C kernels

North (North Mini Code, cohere2_moe.json) is a Cohere2-family MoE: alternating full/sliding attention, a leading dense layer, sigmoid top-8 routing, tied embedding head. It required no new C kernels at all — the circuit composes rmsnorm_forward_llama_production, the pairwise RoPE pair, the flash attention family, group_limited_topk_router_sigmoid_f32, moe_swiglu_expert_forward_q4k_q5k (Q8/Q4_K/Q5_K expert providers from the manifest), and the final_logit_scale_f32 footer added for Command R. The expert kernels themselves are documented in MoE Expert Kernels; the declarative layer plan (no Cohere family branches in lowering or codegen) was the actual work of PR #426.

3. North Micro Vision / Compass: the BF16 frontend

North Micro Vision keeps Cohere2 text semantics and adds a Qwen3-VL-derived vision tower, expressed as three circuits — cohere_compass.json (stitch), cohere_compass_text.json, cohere_compass_vision.json — with BF16 storage throughout. The kernel delta:

patch_projection_image_bf16_native_storage
src/kernels/gemm_kernels_bf16.c:1835. Thread-pooled image→patch embedding projection: temporal-2 weight pair, merge-aligned patch grid, BF16 native storage end to end (no fp32 round-trip between projection and encoder).

gelu_erf_bf16_storage
src/kernels/gelu_kernels.c:609. Exact-erf GELU (not the tanh approximation) on BF16-stored activations — the vision tower's activation contract.

BF16 GEMV/GEMM parallel dispatch.
gemv_bf16 gained gemv_bf16_parallel_dispatch (gemm_kernels_bf16.c:646) — thread-pool row-partitioned BF16 GEMV for decode — and gemm_nt_bf16_bf16_storage gained a parallel dispatch variant (gemm_kernels_bf16.c:1905, +211 lines) with exact BF16 round-trip storage semantics: outputs are rounded to BF16 exactly where the contract says they are.

Evidence (PR #434): the official 577-tensor checkpoint header maps with zero unowned tensors; real-manifest text prefill, text decode, and vision lowering emit zero call-ABI errors. Caveat: public compatibility is not claimed until real-weight image and text generation are certified.

The three-circuit structure above (text extends Cohere2, vision as a component, one declared stitch edge with its own providers) is the general composition mechanism: see Kernel Architecture — Composite Circuits for the anatomy and the measured per-component op/provider breakdown.

4. Cohere Transcribe: the audio stack

Execution status. The cohere_transcribe circuit lowers the frontend and 48-layer encoder into generated C; the 8-layer decoder also executes from a generated model library. A native C host connects these components, manages segment iteration and greedy decoding, and calls native token decoding. The Python native-kernel session remains a diagnostic reference, not a deployment dependency. Its earlier 7.435-second, five-minute, and 42-minute results below are reference-path evidence, not generated-runtime benchmarks. Long audio still requires an externally supplied speech-segment plan. Native VAD, independent timestamp accuracy, multilingual quality, concurrency, and quantized execution remain open or artifact-specific certification gates.

Use the public cstr/cohere-transcribe-03-2026-GGUF F16 artifact for this correctness lane. The official CohereLabs repository is gated; the public GGUF keeps the native smoke reproducible without making private credentials part of the runner.

Historical Python reference path: the following commands reproduce the native-kernel reference session, not the generated-C host.

huggingface-cli download cstr/cohere-transcribe-03-2026-GGUF cohere-transcribe.gguf \
  --local-dir cohere-transcribe-source

python version/v8/scripts/convert_cohere_transcribe_gguf_to_bump_v8.py \
  --gguf cohere-transcribe-source/cohere-transcribe.gguf \
  --output-dir cohere-transcribe-bump

python version/v8/scripts/run_cohere_transcribe_native_v8.py \
  --model cohere-transcribe-bump \
  --audio recording.wav --decode --language en

In the historical reference path, long recordings require validated speech slices because this model fabricates text on silence. Export versioned slices with Silero VAD, then run bounded CKE inference. Each completed slice is atomically retained, so --resume restarts at a slice boundary. The runner emits the raw JSON trajectory, plain text, and SRT captions without hiding decoder loops.

crispasr --vad --vad-model silero --vad-export speech-slices.json \
  --chunk-seconds 30 -f recording.wav

python version/v8/scripts/run_cohere_transcribe_long_audio_v8.py \
  --model cohere-transcribe-bump --audio recording.wav \
  --speech-segments speech-slices.json --language en \
  --output cohere-long.json --text-output transcript.txt --srt-output transcript.srt

Generated-C path: after building the circuit-lowered encoder and decoder runtime bundles, compile the native host and invoke both generated libraries. Each bundle must provide its own libmodel.so, weights.bump, weights_manifest.map, and declared native dependencies. The optional segment plan is a validated native frame-range file; no VAD is performed by the host. Without the plan, the entire WAV is treated as one segment.

make ck-audio-encoder-decoder-transcribe-v8
./build/ck-audio-encoder-decoder-transcribe-v8 \
  /path/to/cohere-encoder/libmodel.so \
  /path/to/cohere-encoder/weights.bump \
  /path/to/cohere-encoder/weights_manifest.map \
  /path/to/cohere-decoder/libmodel.so \
  /path/to/cohere-decoder/weights.bump \
  /path/to/cohere-decoder/weights_manifest.map \
  recording.wav segments.txt

Use export_audio_segment_plan_v8.py to convert an externally produced VAD report into a native segment plan. The generated-path short fixture is checked by certify_cohere_generated_standalone_v8.py; make test-cohere-transcribe-generated-long-audio-auto checks the bounded long-audio path when its runtime, audio, segment, and reference variables are supplied. These do not establish native VAD, independent timestamps, multilingual quality, or general concurrency.

These Python reference-path measurements are not generated-runtime timings. The five-minute certification used a Ryzen 9 9950X3D with 16 threads. The two native runs completed in 67.36 and 66.03 seconds (0.225 and 0.220 real-time factor) with about 8.51 GB peak RSS. The certifier requires the actual engine and audio libraries to appear in the process map, identical artifact/runtime/input/VAD identities, exact repeated token and timestamp trajectories, finite outputs, no repeated multi-word phrase loop, and a bounded oracle WER. These results certify English speech slices up to 30 seconds; they do not certify the VAD model as a CKE kernel or timestamp agreement against an independent forced-alignment oracle. A derived 8 kHz version of the short fixture also reproduces the 16 kHz token trajectory exactly after the existing CKE radius-16 windowed-sinc resampler; broader sample-rate coverage remains open. The compact five-minute certification artifact records the immutable inputs, runtime hashes, checks, and measurements. The complete 42:22 recording then consumed all 123 scheduled speech slices in 639.86 seconds (0.252 real-time factor) at 8.75 GB peak RSS and emitted 729 SRT captions. Against a full CrispASR transcript it differs by one word out of 6,889 (0.0145% WER): CKE preserves a spoken fourfold “it’s,” while CrispASR's generic loop filter retains three. See the compact complete-recording certification artifact. That full run was not repeated, so deterministic long-run evidence remains the two five-minute executions.

All seven live in src/kernels/audio_kernels.c and form the Conformer frontend:

Provider	Source	Contract	Status
`audio_preemphasis_f32`	`audio_kernels.c:303`	pre-emphasis FIR filter, y[t] = x[t] - k*x[t-1]	provider foundation
`audio_stft_power_centered_window_f32`	`audio_kernels.c:498`	centered, windowed STFT power spectrum	provider foundation
`audio_log_mel_time_major_f32`	`audio_kernels.c:553`	log-mel filterbank energies, time-major layout (contract split)	provider foundation
`audio_feature_normalize_per_feature_f32`	`audio_kernels.c:328`	per-mel-bin mean/variance normalization	provider foundation
`audio_conv2d_whc_grouped_f32`	`audio_kernels.c:1025`	grouped conv2d subsampling frontend (WHC layout)	provider foundation
`audio_glu_split_channel_major_f32`	`audio_kernels.c:1115`	GLU split-gating after the frontend conv (how this kernel works)	provider foundation
`audio_relative_shift_f32`	`audio_kernels.c:1173`	relative-position shift for Conformer rel-pos attention (attn.pos_bias_u/v); exactly maps raw[h,q,T-1+k-q] to scores[h,q,k] (how this kernel works)	provider foundation

The model contract (PR #436) validates the real checkpoint generically: a cohere-transcribe GGUF is a 48-layer Conformer encoder (rel-pos attention, depthwise conv + batchnorm, GLU; 39 tensors per block) plus an 8-layer cross-attention decoder (26 tensors per block), with all 2,104 tensors matched exactly. The dedicated converter retains mixed FP16/FP32 storage in a BUMPWGT5 bundle and records every source tensor in the sidecar manifest. PR #441 added the oracle harness version/v8/scripts/certify_cohere_transcribe_oracle_v8.py, which captures stage-wise X-Ray dumps so every encoder block and the generated token trajectory can be compared against the reference. The oracle's unpinned enc_out intermediate is excluded because CrispASR can reuse that buffer before host readback.

click / tap the diagram to expand

The certified audio lane (Whisper, production) is a different stack: Audio Kernels Deep Dive and Whisper Tiny End-to-End.

4.1 The 8-layer cross-attention decoder, kernel by kernel

The decoder is where Transcribe stops looking like Parakeet. Instead of a transducer, it is an 8-layer pre-LayerNorm transformer decoder (width 1024, 8 heads × 128 dims, ReLU FFN 1024→4096→1024) with learned absolute positions, causal self-attention over a growing F16 KV cache, and cross-attention into an immutable F16 encoder K/V cache computed once per speech slice. It composes existing providers end to end — GEMM for every projection, LayerNorm, attention_forward_query_key_head_major_f32_decode_heads for both attention sites, and audio_argmax_first_f32 for the greedy pick — so the decoder needed no new C kernels beyond the encoder family above. The F16 rounding is deliberate and part of the parity story: _f16_cache_round reproduces Cohere's persistent-cache storage contract exactly, for both the growing self-attention cache and the precomputed cross-attention cache.

click / tap the diagram to expand

The kernels inside the encoder that feeds this decoder are walked in the deep dive: Transformer-XL relative attention and the relative shift, the FastConformer convolution module, and the three-way log-mel contract split. The decoder-side attention provider contract is the same head-major decode family the Whisper lane certifies (deep dive, section 9).

5. The pattern

Cohere is the cleanest demonstration of how the registry is supposed to absorb a new family: Command R needed exactly one kernel, North needed zero, Compass needed a BF16 frontend, and only Transcribe — a genuinely new modality with a Conformer frontend — needed a provider family. The cost of a new model is proportional to its contract novelty, not its parameter count, and the fail-closed boundary (provider foundation → certified runtime) is what keeps "we have the kernels" from being read as "we run the model".