Test Report

Automated test results from nightly CI runs. Updated daily at 2am UTC.

About Nightly Tests

The nightly test suite runs automatically at 2am UTC every day. It includes:

• E2E Integration Tests: Full pipeline validation (kernel compile → IR codegen → inference)
  • Kernel compilation (all SIMD variants)
  • IR codegen validation (generates compilable C code)
  • Parallel decode codegen (OpenMP pragmas for v6.6)
  • Inference verification (prompts model, validates coherent response)
• Kernel Unit Tests: Individual kernel parity with PyTorch
• v8 DSL and Codegen Contracts: The aggregate make test-v8-dsl gate is reported as three independently visible nightly rows so a failure identifies the responsible layer: test-v8-dsl-policy enforces zero model/provider hardcoding and map-owned codegen capabilities, test-numerical-contracts validates resolved numerical execution semantics, and test-v8-template-circuit-audit verifies circuit dataflow and lowered bindings.
• v8 Vision Kernel Parity: Vision kernel parity for patch extraction/reconstruction, tiled position embeddings, Qwen3-VL resized position interpolation order, spatial merge/reorder, feature concat/slice, and vision M-RoPE (make test-v8-vision-kernels)
• Qwen3-VL Template/Codegen Contracts: Synthetic GGUF/BF16-independent guards for Qwen3-VL circuit lowering, align-corners position-kernel selection, valid vision M-RoPE width/sections, and bounded semantic tensor exports (make test-v8-qwen3vl)
• Qwen3-VL Methodical Layer Parity: Runs the production circuit in semantic order through position, LayerNorm, Q8 QKV, M-RoPE, attention, Q8 output projection, residual, MLP-up, GELU, MLP-down, and layer output contracts. The nightly row executes production-shape deterministic oracles; make test-qwen3vl-methodical-parity-full requires real artifacts and walks all 27 encoder layers with X-ray, failing at the first divergent edge.
• v8 BF16 Safetensors Lowering: Synthetic BF16 conversion and lowering guard that asserts all vision weights are consumed, align-corners interpolation is selected, and M-RoPE arguments remain valid through generated calls. The target builds both engine and vision libraries first so hardware-independent M-RoPE checks cannot silently skip (make nightly-bf16)
• Vision Encoder Full Accuracy Gate: One local/nightly command runs Qwen3-VL Q8_0 mmproj parity against llama.cpp, then BF16 safetensors parity against PyTorch when the checkpoint and AVX-512 BF16 runtime are available. The gate uses a generated public 1152x896 form-like stress image, validates sparse encoder boundaries (0, 8, 16, 24, 26) plus the final 1008x16384-class prefix, and removes raw tensor dumps by default. Use make vision-encoder-full; set V8_VISION_ENCODER_REQUIRE_ARTIFACTS=1 on provisioned runners, or V8_VISION_ENCODER_FULL_LAYERS=1 for an exhaustive local sweep.
• Scheduled Q4 Vision Encoder Parity: Scheduled GitHub nightly runs download the Qwen3-VL Q8_0 mmproj only when at least 8 GiB RAM and 6 GiB disk are available, then invoke the same vision-encoder-full target. Real BF16 tensor parity remains a periodic AVX-512 BF16 lane; synthetic BF16 conversion/lowering guards continue on every nightly.
• FP16 Attention Contracts: Named numerical guards for llama.cpp-compatible unfused causal prefill and split-KV decode, including FP16 Q/K rounding, FP16 probability/value reduction, FP16 online-softmax partials, FP32 ordered reduction, GQA/padding, the KV 511/512 semantic boundary, and the Qwen3-VL KV=1058, H=32, D=128 shape (make test-attention-f16-split-kv)
• v8 Numerical Reduction Contracts: Circuits request complete attention semantics and executable kernel maps advertise supported reductions; zero or ambiguous providers fail deterministically, while migration tests prove contract validation preserves generated artifacts (make test-numerical-contracts)
• v8 Stitched Dump Alignment: Guards canonical output-projection aliases and preserves absolute CK/llama source-token provenance when mixed-prefix dumps are rebased for per-token comparison (make test-v8-dump-alignment)
• Qwen3-VL Parity Guard Sweep: Local aggregate for GGUF FP16 attention contracts, stitched dump alignment, shared vision kernels, and the BF16 parity lane (make qwen3vl-parity-guards)
• Llama Pairwise RoPE Contract: explicit regression for consecutive-pair RoPE layout selection plus kernel-level pairwise forward coverage and timing (version/v7/scripts/parity/test_check_decode_attention_contract.py, unittest/test_rope.py)
• Sliding-Window Contract: Explicit sliding attention prefill/decode contract test (unittest/test_attention_sliding_contract.py)
• BF16 Tests: BFloat16 precision tests (when CPU supports it)
• Quantization Tests: Q4_K, Q6_K quantized kernel tests
• v7 Kernel Map Contracts: sync + validator + contract tests for the active v7 map surface (make v7-kernel-map-contracts)
• v7 Backprop Kernel Parity: Optimizer + QK-norm backward ISA matrix + RMSNorm/SwiGLU backward + GEMM backward sweep (make v7-kernel-parity-train)
• v7 Training Family Regression: family-level A/B/C/D/E/F training regimen across the full decoder family matrix (make regression-training-full nightly, make regression-training-fast for local fast coverage)
• v7 IR Visualizer E2E: Runbook-path regression with explicit run-dir wiring, decode profile checks, and tiny train-runtime ASan artifact validation (make visualizer, make visualizer-full)
• v7 Visualizer Health Gate: Contract-driven static analysis (160 checks) + JS unit tests (100 test vectors) for IR visualizer, dataset viewer, and IR hub (make v7-visualizer-health)
• v8 Inference Regression: Family bring-up gate for Gemma3, Qwen2, Qwen3, Qwen3.5, and Nanbeige with build/smoke/coherence/contract checks (make regression-fast, aliasing the promoted make v8-regression-fast lane)
• Architecture Contract Dashboard: v8 template/circuit, lowered IR dataflow, generated-C preservation, runtime-path equivalence, and model contract coverage summary (make test-architecture-contracts)
• v8 Qwen3-VL Vision Smoke: Cached image → mmproj → decoder smoke for the 8B multimodal path; skips cleanly when the runner does not have the large artifacts (make test-v8-qwen3vl-e2e-smoke)
• v8 Gemma4 Vision Smoke: Cached Gemma4 Q4_K_M decoder plus mmproj-F16.gguf image bridge smoke; gated by high-memory and local artifact checks (make test-v8-gemma4-vision-smoke)
• v8 High-Memory Model Smokes: Gemma4, Nemotron Nano 9B, and GLM-4 9B run as explicit inference lanes and skip when the runner does not have enough RAM (make test-v8-gemma4-highmem, make test-v8-nemotron9-highmem, make test-v8-glm4-highmem)
• v7 Regression Ledger: Canonical root-cause log for fixed and monitored bugs, consumed by operators in the visualizer (version/v7/reports/REGRESSION_LEDGER.md, version/v7/reports/REGRESSION_LEDGER.json)
• llama.cpp Parity: CK vs llama.cpp kernel parity flow (scripts/run_parity_smoketest.sh, make llamacpp-parity-full)
• v6.6 Preflight Gates: Tooling contracts + 3-model matrix validation (make v6.6-validate-contracts, make v6.6-validate-matrix)
• Integration Tests: Full layer and model parity

View the GitHub Actions workflow for detailed logs.

See also: Version History for project roadmap and milestone tracking.

Visualizer Health Matrix

Contract-driven test coverage for the IR Visualizer and Dataset Viewer. Each component declares its tab/function/DOM contracts in JSON; tests validate against those contracts at every push. v7 owns the promoted training/backprop visualizer lane; v8 owns the inference and multimodal observability lane with its own IR report, run hub, dataset viewer, and dedicated vision artifact coverage. See the full test runbook.

Run locally: make v7-visualizer-health · make v8-visualizer-health · make v8-visualizer-vision-artifacts

IR Visualizer — 11 tabs, 19 core functions

L2-tested utilities: formatBytes (4) · normalizeShapeInput (7) · formatShapeDisplay (3) · normalizeMode (3) · escapeHtml (3) · quoteShell (3) · normalizePathString (2) · pathDirname (3) · extractGgufStem (3) · relativePathFromTo (4) — 35 tests

Dataset Viewer — 12 tabs, 30 core functions

L2-tested functions: attnColor (9) · embColor (3) · cosineSim (4) · attnEntropy (4) · avgMatrices (3) · embNormalise (4) — 27 tests

Coverage Summary

Level 3 — Generated-File E2E (Nightly)

Validates the full generation → validation chain: generate all three visualizer HTML files from the latest run, then run L1 health checks + embedded JSON structure + cross-artifact consistency on the output. The v7 version uses a training-capable run; the v8 version uses the v8 inference cache and hub/tool wrappers.

Run locally: make v7-visualizer-generated-e2e · make v8-visualizer-generated-e2e · Specific run: make v7-visualizer-generated-e2e RUN=/path/to/run