v8 Inference Runbook

Modality gates

Long-context comparison does not require storing every activation from every token. It requires deterministic input hashes, complete provider identity, selected layer/state checkpoints at fixed positions, final logits, and a matched decode trajectory. Any first divergence can then be expanded into a full operation-level X-Ray capture.

One command before a demo

Run the capacity-aware compatibility sweep before leaving the machine overnight. It exercises the promoted decoder families, compiler and memory-plan contracts, llama.cpp numerical gates, Instella, audio, vision, configured high-memory models, and both Qwen private OCR corpora.

make nightly-demo-readiness

In the morning, open build/demo-readiness/nightly-latest.md. The machine-readable evidence is written to build/demo-readiness/nightly-latest.json. A configured lane that breaks is a failure; an optional model or private corpus that is not configured is shown explicitly as SKIP.

Private OCR output is redacted from the aggregate report. Configure Qwen3-VL with CK_QWEN3VL_OCR_MANIFEST, configure Qwen3.6-VL with CK_QWEN36VL_OCR_MANIFEST, CK_QWEN36VL_DECODER_GGUF, and CK_QWEN36VL_ENCODER_RUNTIME, and point CK_LLAMA_CPP_ROOT at the pinned llama.cpp oracle. Existing completed corpus cases resume instead of being repeated.

Long-context capacity and quality sweep

Run the resumable long-context matrix separately from demo readiness. It visits every configured family at 2K, 8K, 32K, 64K, 128K, and 256K tokens, where supported. The 64K, 128K, and 256K rows are the long-context certification region. Every row records exact consumed-token evidence, input hashes, prefill/decode wall time, average active cores, peak RSS, first-logit repeatability, selected generated providers, and matched llama.cpp performance when a local GGUF oracle is available. By default, a CKE performance row fails when average occupancy is below half of the requested worker count; override --min-active-core-fraction only for a documented topology experiment.

V8_LONG_CONTEXT_ARGS="--allow-download --threads 16" \
make certify-v8-long-context

Run the engineering-quality phase independently when you need reviewable outputs before the multi-day capacity ladder finishes. Every model receives the same complete FP32 RoPE task. Its extracted C source is checked with a strict syntax-only compiler invocation, then passed verbatim back to the model to produce a matching standalone SVG. Generated code is never executed automatically. The gate proves artifact structure, not algorithmic correctness; human review and a numerical oracle remain mandatory before a generated kernel can enter CKE.

V8_ENGINEERING_QUALITY_ARGS="--allow-download --threads 16" \
make certify-v8-engineering-quality

To certify a full 128K generation envelope, allocate the total context explicitly. Here --quality-context 8192 reserves room for the prompt and --quality-total-context 131072 is the exact compiled runtime capacity. Decode receives every remaining token after the reserve and safety margin, and normally stops on the model's EOS signal. The report records the allocated context, effective output budget, consumed tokens, and native stop reason; it does not relabel an 8K+8K run as 128K.

V8_ENGINEERING_QUALITY_ARGS="--allow-download --threads 0 --cpu-policy physical --quality-context 8192 --quality-total-context 131072" \
make certify-v8-engineering-quality

Configure local checkpoints with V8_QWEN36_MODEL, V8_QWEN38_MODEL, V8_QWEN35_MOE_MODEL, V8_GEMMA4_MODEL, V8_GLM4_MODEL, V8_NEMOTRON9_MODEL, V8_INSTELLA_MODEL, V8_KIMI_MODEL, V8_LAGUNA_MODEL, and V8_COHERE2_MODEL. The corresponding V8_*_GGUF variables may identify a separate llama.cpp oracle. Reports are written incrementally to build/v8-long-context-certification/summary.md and summary.json, so an interrupted multi-day run resumes completed rows. Qwen3.6-VL is intentionally excluded from this text matrix and remains covered by make test-qwen36vl-private-corpus-parity-auto, which exercises its encoder, bridge, decoder, and 40-image OCR corpus.

The capacity matrix uses a deterministic fixed-token sequence so exact consumption, hashes, timing, and oracle comparisons remain reproducible. It does not claim that normal prompts should fill every supported context window. The standalone engineering command writes to build/v8-engineering-quality/quality/: open index.html for the readable gallery, or inspect each model's exact prompt, raw response, clean .c/.svg artifact, compiler logs, trace, and validation JSON. Repository, documentation, retrieval, and multi-turn application corpora are the next certification layer.

How The Vision Side Works

If you want the architecture view instead of the operator commands, read v8 Vision Encoder Architecture. That page explains how the encoder graph is derived from GGUF + template + lowering and how the bridge hands the prefix into the decoder.

For parity-specific investigation, see Vision Encoder Parity.

How The Audio Side Works

Whisper Tiny transcription runs through the same circuit and kernel-contract path: generated FP32 encoder and decoder from PCM16 WAV, with nightly audio gates and an opt-in end-to-end artifact test. See Whisper Tiny End-to-End for the pipeline diagram, usage, evidence, and current limitations.

Use this path when you want inference-only runs without training a local model.

These raw CLI examples use the shell wrapper so a first-time user can be prompted to create .venv and install version/v8/requirements.txt if the repo-local environment is missing.

Qwen3.8 27B Dense Q4_K_M

Run from the repository root. This command uses the standard GGUF, not an Unsloth UD mixed-quant artifact.

version/v8/scripts/cks-v8-run run \
  hf://ggml-org/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q4_K_M.gguf \
  --context-len 16384 \
  --force-convert --force-compile \
  --chat-template auto \
  --prompt 'Give me a detailed example of C, Python and SQL code.' \
  --max-tokens 256 \
  --temperature 0.0

16384 is context capacity, not consumed prompt length. This is a short generation check; allocate only what fits available RAM.

Qwen3.8 Flash Next Q4_K_M

Experimental text inference. The example uses the local GGUF path on the 160 GB Ryzen test node. On another machine, replace that path with your matching local artifact. The tested file is 119,150,722,112 bytes; this command does not download it or certify other Flash quantizations.

version/v8/scripts/cks-v8-run run \
  "$HOME/.cache/cke-nvme/qwen38-flash-next/gguf/qwen3.8-flash-next-Q4_K_M.gguf" \
  --context-len 8192 \
  --force-convert --force-compile \
  --chat-template auto \
  --prompt 'Give me a detailed example of C, Python and SQL code.' \
  --max-tokens 256 \
  --temperature 0.0

The Flash example is for a high-memory host, not the 64 GB P3. 8192 is capacity, not an 8K-input certification. See the evidence table above for numerical and long-context limits.

Gemma 3 270M

version/v8/scripts/cks-v8-run run \
  hf://unsloth/gemma-3-270m-it-GGUF/gemma-3-270m-it-Q5_K_M.gguf \
  --context-len 1024 --force-compile --force-convert --chat-template=auto \
  --generate-visualizer

Gemma 4 E4B IT

.venv/bin/python version/v8/scripts/ck_run_v8.py run \
  hf://unsloth/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf \
  --context-len 2048 \
  --force-convert --force-compile \
  --prompt 'Give me a detailed example of C code.' \
  --chat-template gemma4 \
  --max-tokens 1024 \
  --temperature 0.0

Use this as the current Gemma4 GGUF coherence smoke. A healthy run should produce a normal long answer and stop by EOS or --max-tokens, not by prompt-marker echo or language corruption. The tested local Q4_K_M run generated 1024 coherent C-code tokens; performance work remains separate.

Gemma 4 Assistant / MTP Drafter

.venv/bin/python version/v8/scripts/ck_run_v8.py run \
  hf://google/gemma-4-E4B-it-assistant \
  --run /tmp/ck-gemma4-assistant-runtime \
  --context-len 1024 \
  --force-convert --force-compile \
  --generate-only \
  --chat-template none \
  --allow-raw-prompt

The assistant artifact is a tiny speculative/MTP drafter, not a standalone chat model. A healthy bring-up converts safetensors directly to BUMP, lowers q-only shared-KV attention, generates C, and compiles libmodel.so. Real chat speedup comes only after pairing this runtime with a compatible Gemma4 backbone via --speculative-draft-model-dir.

Nemotron Nano 9B v2

.venv/bin/python version/v8/scripts/ck_run_v8.py run \
  hf://bartowski/nvidia_NVIDIA-Nemotron-Nano-9B-v2-GGUF/nvidia_NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf \
  --context-len 2048 \
  --force-convert --force-compile \
  --prompt 'Give me a concise example of C code.' \
  --chat-template auto \
  --max-tokens 256 \
  --temperature 0.0

Use this as the current Nemotron-H / Mamba2 GGUF smoke. Native BPE text encode is enabled by default for generated BPE runtimes; set CK_DISABLE_FULL_BPE_TOKENIZER=1 only when you intentionally want token-id prompts plus lightweight token display. A healthy run should produce normal instructional text, not token-zero collapse or repeated punctuation.

GLM4

version/v8/scripts/cks-v8-run run \
  hf://unsloth/GLM-4-9B-0414-GGUF/GLM-4-9B-0414-Q4_K_M.gguf \
  --context-len 1034 \
  --force-convert --force-compile \
  --chat-template glm4 \
  --prompt 'Give me a detailed example of C, Python and SQL code.' \
  --max-tokens 256 \
  --temperature 0.0

GLM4 has a real Q4_K_M runtime lane with partial pairwise RoPE, persistent FP16 KV cache, and llama.cpp parity coverage. --force-compile force-builds both libckernel_engine.so and libckernel_tokenizer.so, compiles the generated model into a staging directory, verifies that the complete bundle loads without unresolved symbols, and only then replaces the cached runtime. No manual library copy is required.

Kimi-VL A3B Text Decoder

.venv/bin/python version/v8/scripts/ck_run_v8.py run \
  /path/to/moonshotai--Kimi-VL-A3B-Instruct \
  --run /srv/cke/profiles/kimi-vl-a3b-instruct \
  --context-len 2048 \
  --force-convert --force-compile \
  --prompt 'Give me a concise example of safe C code.' \
  --chat-template kimi_vl \
  --max-tokens 64 \
  --temperature 0.0

# Capacity-aware nightly smoke using the same source directory:
V8_KIMI_MODEL=/path/to/moonshotai--Kimi-VL-A3B-Instruct \
make test-v8-kimi-highmem

The official 32.8 GB sharded checkpoint now converts all text-decoder tensors, preserves the embedded TikToken protocol, and runs coherently on a 64 GB host. Three repeated Ryzen runs produced the same 32-token response hash. X-Ray matched PyTorch's first token and complete top-20 ranking, but internal BF16 drift becomes material around layer 7, so this is text E2E certification rather than bit-exact internal parity. MoonViT and the multimodal bridge remain outside the certified path. See v8 MLA / Kimi Decode Cache for the architecture notes.

Laguna-XS 2.1

.venv/bin/python version/v8/scripts/ck_run_v8.py run \
  /path/to/Laguna-XS-2.1-Q4_K_M.gguf \
  --run /srv/cke/profiles/laguna-xs-2.1 \
  --context-len 2048 \
  --force-convert --force-compile \
  --prompt 'Write a complete C program that prints the first ten Fibonacci numbers.' \
  --chat-template auto \
  --thinking-mode suppressed \
  --max-tokens 512 \
  --temperature 0.0

The tested 20.27 GB Q4_K_M artifact converts all 684 mapped entries and produces coherent text and compilable C on Ryzen. The circuit declares full versus sliding attention, RoPE width, softplus attention gating, and mixed Q4/Q6 expert providers without a Laguna branch in compiler or code generation. Visible thinking is supported but can consume more than 1,024 tokens before the final answer; use a larger output budget when testing that mode. This is coherent E2E bring-up with strong leaf evidence, not long-trajectory bit-exact certification.

Cohere2 Command R

version/v8/scripts/cks-v8-run run \
  /path/to/command-r7b-q4_k_m.gguf \
  --context-len 2048 --force-compile --force-convert --chat-template=auto \
  --max-tokens 64 --temperature 0.0

Cohere2 needs no family-specific flag: the GGUF model map routes it to the cohere2 circuit automatically. Contract test: python -m unittest tests.test_v8_cohere2_contract.

Qwen2 0.5B Instruct

version/v8/scripts/cks-v8-run run \
  hf://Qwen/Qwen2-0.5B-Instruct-GGUF/qwen2-0_5b-instruct-q4_k_m.gguf \
  --context-len 1024 --force-compile --force-convert \
  --generate-visualizer

Qwen3 0.6B

version/v8/scripts/cks-v8-run run \
  hf://Qwen/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q8_0.gguf \
  --context-len 1024 \
  --force-convert --force-compile \
  --generate-visualizer

Qwen3.5 0.8B

python3 version/v8/scripts/ck_run_v8.py run \
  hf://unsloth/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q4_K_M.gguf \
  --force-convert --force-compile \
  --context-len 1034

The canonical v8 bring-up path is the hf://... URI. That materializes the run under ${CK_CACHE_DIR:-$HOME/.cache/ck-engine-v8/models}/unsloth--Qwen3.5-0.8B-GGUF. Local GGUF paths are still supported when you intentionally want an offline or copied artifact.

NaanBeige 4.1 3B

version/v8/scripts/cks-v8-run run \
  hf://mradermacher/Nanbeige4.1-3B-GGUF/Nanbeige4.1-3B.Q4_K_M.gguf \
  --context-len 1024 --force-compile --force-convert \
  --chat-template auto \
  --generate-visualizer

Current scope: these are the validated v8 text-family operator surfaces. The multimodal Qwen3-VL path remains the promoted vision baseline; Gemma4 vision now has a high-memory bridge smoke lane for second-family coverage.

Confirm Kernel Dispatch

Startup evidence should include a line similar to:

[CK parallel prefill] Prepared 184 AVX-512 VNNI x16 Q4_K weights at load time

With CK_DEBUG_Q4K_PREFILL_DISPATCH=1, the projection diagnostics should show the x16_mreuse provider for the eligible Q4 prefill shapes.

This does not mean every CPU selects AVX-512: AVX-512/VNNI-capable systems may select the x16 provider, while other systems use a certified AVX2 or reference fallback. Kernel selection is intended to become automatic from CPU capability, model metadata, quantization format, and runtime shape; the debug environment variables are diagnostic overrides, not a public configuration surface.

What End-to-End Certification Means

1. Tokenizer parity: generated C encode/decode agrees with the declared Hugging Face or reference tokenizer.
2. Model parity: identical token IDs produce the required logits or generated token trajectory against the numerical oracle.
3. Native product parity: ck-cli-v8 performs generated chat formatting, C tokenization, model execution, generated stop policy, and C detokenization without Python in the request path.

Headless Server Access

When the server has no window manager, serve the canonical model root and use an SSH tunnel. Keeping the HTTP server on loopback avoids exposing unauthenticated reports to the surrounding network.

# Run on the headless CKE server.
cd "${CK_CACHE_DIR:-$HOME/.cache/ck-engine-v8/models}"
python3 -m http.server 7021 --bind 127.0.0.1

# Run on the workstation that has a browser.
ssh -L 7021:127.0.0.1:7021 USER@SERVER

# Then open:
http://127.0.0.1:7021/ir_hub.html

On a trusted LAN or Tailscale network, binding with --bind 0.0.0.0 makes the hub reachable at http://SERVER_IP:7021/ir_hub.html. Python's development server provides no authentication or TLS; do not expose it directly to the public internet.

Use these gates when changing v8 visualizer tabs, embedded JSON, run-hub discovery, dataset viewer contracts, or multimodal bridge artifacts. They are intentionally fast enough to run locally before a PR.

# Source-level tab/function/DOM contracts plus pure JS utility checks.
make v8-visualizer-health

# Generate and validate ir_report.html, dataset_viewer.html, and ir_hub_v8.html.
make v8-visualizer-generated-e2e

# Validate encoder + bridge + decoder visualizer rendering for a synthetic vision run.
make v8-visualizer-vision-artifacts

v7 still owns the promoted training/backprop visualizer lane. v8 now owns inference and multimodal observability: lowered decoder graphs, generated C metadata, logical memory, kernel-flow summaries, bridge reports, and vision-prefix dataflow.