Test Report

Automated test results from nightly CI runs. Updated daily at 2am UTC.

About Nightly Tests

The nightly test suite runs automatically at 2am UTC every day. It includes:

• E2E Integration Tests: Full pipeline validation (kernel compile → IR codegen → inference)
  • Kernel compilation (all SIMD variants)
  • IR codegen validation (generates compilable C code)
  • Parallel decode codegen (OpenMP pragmas for v6.6)
  • Inference verification (prompts model, validates coherent response)
• Kernel Unit Tests: Individual kernel parity with PyTorch
• v8 DSL and Codegen Contracts: The aggregate make test-v8-dsl gate is reported as three independently visible nightly rows so a failure identifies the responsible layer: test-v8-dsl-policy enforces zero model/provider hardcoding and map-owned codegen capabilities, test-numerical-contracts validates resolved numerical execution semantics, and test-v8-template-circuit-audit verifies circuit dataflow and lowered bindings.
• v8 Vision Kernel Parity: Vision kernel parity for patch extraction/reconstruction, tiled position embeddings, Qwen3-VL resized position interpolation order, spatial merge/reorder, feature concat/slice, and vision M-RoPE (make test-v8-vision-kernels)
• Qwen3-VL Template/Codegen Contracts: Synthetic GGUF/BF16-independent guards for Qwen3-VL circuit lowering, align-corners position-kernel selection, valid vision M-RoPE width/sections, and bounded semantic tensor exports (make test-v8-qwen3vl)
• Qwen3-VL Methodical Layer Parity: Runs the production circuit in semantic order through position, LayerNorm, Q8 QKV, M-RoPE, attention, Q8 output projection, residual, MLP-up, GELU, MLP-down, and layer output contracts. The nightly row executes production-shape deterministic oracles; make test-qwen3vl-methodical-parity-full requires real artifacts and walks all 27 encoder layers with X-ray, failing at the first divergent edge.
• v8 BF16 Safetensors Lowering: Synthetic BF16 conversion and lowering guard that asserts all vision weights are consumed, align-corners interpolation is selected, and M-RoPE arguments remain valid through generated calls. The target builds both engine and vision libraries first so hardware-independent M-RoPE checks cannot silently skip (make nightly-bf16)
• Vision Encoder Full Accuracy Gate: One local/nightly command runs Qwen3-VL Q8_0 mmproj parity against llama.cpp, then BF16 safetensors parity against PyTorch when the checkpoint and AVX-512 BF16 runtime are available. The gate uses a generated public 1152x896 form-like stress image, validates sparse encoder boundaries (0, 8, 16, 24, 26) plus the final 1008x16384-class prefix, and removes raw tensor dumps by default. Use make vision-encoder-full; set V8_VISION_ENCODER_REQUIRE_ARTIFACTS=1 on provisioned runners, or V8_VISION_ENCODER_FULL_LAYERS=1 for an exhaustive local sweep.
• Scheduled Q4 Vision Encoder Parity: Scheduled GitHub nightly runs download the Qwen3-VL Q8_0 mmproj only when at least 8 GiB RAM and 6 GiB disk are available, then invoke the same vision-encoder-full target. Real BF16 tensor parity remains a periodic AVX-512 BF16 lane; synthetic BF16 conversion/lowering guards continue on every nightly.
• FP16 Attention Contracts: Named numerical guards for llama.cpp-compatible unfused causal prefill and split-KV decode, including FP16 Q/K rounding, FP16 probability/value reduction, FP16 online-softmax partials, FP32 ordered reduction, GQA/padding, the KV 511/512 semantic boundary, and the Qwen3-VL KV=1058, H=32, D=128 shape (make test-attention-f16-split-kv)
• v8 Numerical Reduction Contracts: Circuits request complete attention semantics and executable kernel maps advertise supported reductions; zero or ambiguous providers fail deterministically, while migration tests prove contract validation preserves generated artifacts (make test-numerical-contracts)
• v8 Stitched Dump Alignment: Guards canonical output-projection aliases and preserves absolute CK/llama source-token provenance when mixed-prefix dumps are rebased for per-token comparison (make test-v8-dump-alignment)
• Qwen3-VL Parity Guard Sweep: Local aggregate for GGUF FP16 attention contracts, stitched dump alignment, shared vision kernels, and the BF16 parity lane (make qwen3vl-parity-guards)
• Llama Pairwise RoPE Contract: explicit regression for consecutive-pair RoPE layout selection plus kernel-level pairwise forward coverage and timing (version/v7/scripts/parity/test_check_decode_attention_contract.py, unittest/test_rope.py)
• Sliding-Window Contract: Explicit sliding attention prefill/decode contract test (unittest/test_attention_sliding_contract.py)
• BF16 Tests: BFloat16 precision tests (when CPU supports it)
• Quantization Tests: Q4_K, Q6_K quantized kernel tests
• v7 Kernel Map Contracts: sync + validator + contract tests for the active v7 map surface (make v7-kernel-map-contracts)
• v8 Kernel Allocation Debt: scans every C source under src/kernels for allocator and free calls, classifies production, frontend, oracle, and test debt, rejects new call sites or count increases, and publishes remaining mapped providers with scratch ownership status (make test-v8-kernel-allocations)
• v7 Backprop Kernel Parity: Optimizer + QK-norm backward ISA matrix + RMSNorm/SwiGLU backward + GEMM backward sweep (make v7-kernel-parity-train)
• v7 Training Family Regression: family-level A/B/C/D/E/F training regimen across the full decoder family matrix (make regression-training-full nightly, make regression-training-fast for local fast coverage)
• v7 IR Visualizer E2E: Runbook-path regression with explicit run-dir wiring, decode profile checks, and tiny train-runtime ASan artifact validation (make visualizer, make visualizer-full)
• v7 Visualizer Health Gate: Contract-driven static analysis (160 checks) + JS unit tests (100 test vectors) for IR visualizer, dataset viewer, and IR hub (make v7-visualizer-health)
• v8 Inference Regression: Family bring-up gate for Gemma3, Qwen2, Qwen3, Qwen3.5, and Nanbeige with build/smoke/coherence/contract checks (make regression-fast, aliasing the promoted make v8-regression-fast lane)
• Architecture Contract Dashboard: v8 template/circuit, lowered IR dataflow, generated-C preservation, runtime-path equivalence, and model contract coverage summary (make test-architecture-contracts)
• v8 Qwen3-VL Vision Smoke: Cached image → mmproj → decoder smoke for the 8B multimodal path; skips cleanly when the runner does not have the large artifacts (make test-v8-qwen3vl-e2e-smoke)
• v8 Gemma4 Vision Smoke: Cached Gemma4 Q4_K_M decoder plus mmproj-F16.gguf image bridge smoke; gated by high-memory and local artifact checks (make test-v8-gemma4-vision-smoke)
• v8 High-Memory Model Smokes: Gemma4, Nemotron Nano 9B, and GLM-4 9B run as explicit inference lanes and skip when the runner does not have enough RAM (make test-v8-gemma4-highmem, make test-v8-nemotron9-highmem, make test-v8-glm4-highmem)
• v7 Regression Ledger: Canonical root-cause log for fixed and monitored bugs, consumed by operators in the visualizer (version/v7/reports/REGRESSION_LEDGER.md, version/v7/reports/REGRESSION_LEDGER.json)
• llama.cpp Parity: CK vs llama.cpp kernel parity flow (scripts/run_parity_smoketest.sh, make llamacpp-parity-full)
• v6.6 Preflight Gates: Tooling contracts + 3-model matrix validation (make v6.6-validate-contracts, make v6.6-validate-matrix)
• Integration Tests: Full layer and model parity

View the GitHub Actions workflow for detailed logs.

See also: Version History for project roadmap and milestone tracking.

Visualizer Health Matrix

Contract-driven test coverage for the IR Visualizer and Dataset Viewer. Each component declares its tab/function/DOM contracts in JSON; tests validate against those contracts at every push. v7 owns the promoted training/backprop visualizer lane; v8 owns the inference and multimodal observability lane with its own IR report, run hub, dataset viewer, and dedicated vision artifact coverage. See the full test runbook.

Run locally: make v7-visualizer-health · make v8-visualizer-health · make v8-visualizer-vision-artifacts

IR Visualizer — 11 tabs, 19 core functions

L2-tested utilities: formatBytes (4) · normalizeShapeInput (7) · formatShapeDisplay (3) · normalizeMode (3) · escapeHtml (3) · quoteShell (3) · normalizePathString (2) · pathDirname (3) · extractGgufStem (3) · relativePathFromTo (4) — 35 tests

Dataset Viewer — 12 tabs, 30 core functions

L2-tested functions: attnColor (9) · embColor (3) · cosineSim (4) · attnEntropy (4) · avgMatrices (3) · embNormalise (4) — 27 tests

Coverage Summary

Level 3 — Generated-File E2E (Nightly)

Validates the full generation → validation chain: generate all three visualizer HTML files from the latest run, then run L1 health checks + embedded JSON structure + cross-artifact consistency on the output. The v7 version uses a training-capable run; the v8 version uses the v8 inference cache and hub/tool wrappers.

Run locally: make v7-visualizer-generated-e2e · make v8-visualizer-generated-e2e · Specific run: make v7-visualizer-generated-e2e RUN=/path/to/run