v7 Backprop IR Pipeline (Operator Guide)

This page explains how v7 training works end to end, with a focus on what is generated from IR, what is layout/codegen output, where logic lives, and how to debug correctness quickly.

What Runs in C vs Python

Mental Model: What Is Smart and What Is Dumb

From Init to Train

1. Initialize run: choose init policy, dimensions, dtype policy, and template.
2. Emit weights: write weights.bump and weights_manifest.json.
3. Build IR1: forward graph and typed tensor registry.
4. Lower IR2: backward graph from IR1 + grad rules, with explicit accumulations.
5. Generate layout: contiguous memory map with sections and canary ranges.
6. Codegen runtime: emit train runtime C and compile to libtrain.so.
7. Execute train: run CK path, optional PyTorch oracle checks, write reports.

Drift Timeline: Early Schedule Bug to CE Numeric Tightening

Yes, execution order changed in an important way. The generated train runtime now enforces accumulation-window semantics so optimizer updates only happen at the true boundary of CK_GRAD_ACCUM_STEPS micro-steps.

• Before: optimizer timing could drift relative to the intended accumulation window.
• Now: each window does zero_grad once, accumulates grads over micro-steps, then applies one optimizer step at boundary.
• Impact: early drift signal (~step 65 in stress) was removed; remaining drift moved to much later horizons and required CE semantic/numeric tightening.

if g_accum_step == 0: zero_grad()
forward()
backward()
g_accum_step += 1
if g_accum_step >= CK_GRAD_ACCUM_STEPS:
    scale_grads(1 / g_accum_step)  # avg over accumulation window
    optimizer_step()
    g_accum_step = 0

# Built-in
version/v7/scripts/cks-v7-run init \
  --run /tmp/v7_exp1 \
  --template qwen3 \
  --init xavier_uniform \
  --generate-ir --generate-runtime --strict

# Custom template
version/v7/scripts/cks-v7-run init \
  --run /tmp/v7_exp_custom \
  --template my_arch \
  --template-file /absolute/path/to/my_template.json \
  --init xavier_uniform \
  --generate-ir --generate-runtime --strict

IR1, IR2, Layout, Codegen Responsibilities

Why Train Builder Is Separate From Inference Builder (For Now)

• Inference IR is decode/prefill-oriented and optimized around quantized runtime contracts.
• Training IR adds backward-only semantics: grad fanout merges, loss seed ops, optimizer state, clip/update boundaries.
• Training layout needs persistent regions inference does not own (grad.weight.*, optimizer.m.*, optimizer.v.*).
• Current policy: keep inference pipeline stable, iterate training pipeline quickly, then unify once parity + diagnostics are robust.

So today: inference uses build_ir_v7.py; training uses build_ir_train_v7.py + lower_ir2_backward_v7.py. Both share IR types and kernel registry contracts.

How Residual, GQA, and Splits Stitch Backward

The forward graph can branch. IR2 makes the reverse merge explicit by emitting accumulation ops where gradients meet.

Memory Layout and Canary Diagnostics

Training layout is contiguous and sectioned. Canary guards and runtime checks detect out-of-bounds writes and readonly violations.

Runbook: End-to-End Commands

version/v7/scripts/cks-v7-run init \
  --run /tmp/v7_exp1 \
  --template qwen3 \
  --init xavier_uniform \
  --layers 2 --embed-dim 128 --hidden-dim 256 \
  --num-heads 8 --num-kv-heads 4 \
  --generate-ir --generate-runtime --strict

version/v7/scripts/cks-v7-run train \
  --run /tmp/v7_exp1 \
  --backend ck \
  --prompt "hello" \
  --train-epochs 3 --train-seq-len 16 --train-total-tokens 1024 --train-grad-accum 8 \
  --train-strict

version/v7/scripts/cks-v7-run train \
  --run /tmp/v7_exp1 \
  --backend both \
  --prompt "hello" \
  --train-epochs 1 --train-seq-len 16 --train-total-tokens 1024 --train-grad-accum 8 \
  --parity-on --parity-profile balanced --dump-on-drift --drift-topk 8

python3 version/v7/tools/open_ir_visualizer.py \
  --generate --run /tmp/v7_exp1 --html-only

New: One-Command Data -> Tokenizer -> Train Pipeline

For operator workflows, use the pipeline wrapper to avoid manual wiring errors such as missing data files or token/vocab mismatches.

make v7-train-data-pipeline \
  RUN=/tmp/v7_pipeline_byte \
  V7_PIPELINE_TOKENIZER=byte \
  V7_PIPELINE_WITH_TORCH=1

make v7-train-data-pipeline \
  RUN=/tmp/v7_pipeline_bpe \
  V7_PIPELINE_TOKENIZER=bpe \
  V7_PIPELINE_WITH_TORCH=0

New: Runtime Stitch Graph in IR Visualizer

The Training tab now includes a function-level runtime graph fed by train_exec_plan.json with explicit stages:

• forward ops
• backward ops
• gradient accumulation ops
• optimizer/update transition

Use this graph when first loss is correct but later updates diverge. It makes boundary mistakes visible at function level without opening generated C manually.

Verification Matrix: Epoch 1 to 10 + Checkpoints + Oracle

# 0) init
.venv/bin/python version/v7/scripts/ck_run_v7.py init \
  --run /tmp/v7_oracle_10ep \
  --init xavier_uniform \
  --layers 2 --vocab-size 256 --embed-dim 128 --hidden-dim 256 \
  --num-heads 8 --num-kv-heads 4 --context-len 128 \
  --generate-ir --generate-runtime --strict

# 1) epoch-1 strict oracle (every step) + replay + periodic checkpoints
.venv/bin/python version/v7/scripts/ck_run_v7.py train \
  --run /tmp/v7_oracle_10ep \
  --backend ck \
  --train-epochs 1 --train-seq-len 8 --train-total-tokens 64 --train-grad-accum 2 \
  --train-vocab 256 --train-d-model 64 --train-hidden 128 \
  --parity-on --oracle pytorch --parity-profile debug --parity-every 1 \
  --parity-replay-on-check --train-save-every 4

# 2) epoch-10 sweep strict oracle + replay + checkpoint cadence
.venv/bin/python version/v7/scripts/ck_run_v7.py train \
  --run /tmp/v7_oracle_10ep \
  --backend ck \
  --train-epochs 10 --train-seq-len 8 --train-total-tokens 64 --train-grad-accum 2 \
  --train-vocab 256 --train-d-model 64 --train-hidden 128 \
  --parity-on --oracle pytorch --parity-profile balanced --parity-every 1 \
  --parity-replay-on-check --train-save-every 20

# 3) memory verification suite (canary toggle + fault injection + ASan agreement)
.venv/bin/python version/v7/scripts/ck_run_v7.py train \
  --run /tmp/v7_oracle_10ep \
  --backend ck \
  --train-epochs 1 --train-seq-len 8 --train-total-tokens 64 --train-grad-accum 2 \
  --train-vocab 256 --train-d-model 64 --train-hidden 128 \
  --train-verify-memory --train-verify-steps 4

Backprop + PyTorch Parity Test Suite

This is the concrete v7 inventory used to keep backprop parity green. By default, artifacts write to version/v7/.cache/reports/ (override with V7_REPORT_DIR).

Exhaustive Test-Case Matrix (Backprop + Parity)

PR4.5 Throughput Track

• Add explicit threaded GEMM dispatch for v7 training runtime (not serial-only blocked calls).
• Compile runtime with consistent parallel flags and deterministic thread control.
• Prepack/plan GEMM weight access to reduce cache misses.
• Reduce backward buffer traffic hotspots (grad_accumulate, memset, memmove).
• Keep strict parity tolerances unchanged while optimizing.

Performance Workflow (After Parity Is Green)

1. Run CK baseline without oracle: --backend ck --parity-on=false.
2. Run reference harness timing: --backend both (captures CK vs torch step metrics).
3. Use make profile-v7-full or make v7-perf-gate for perf + flamegraph artifacts (see v7-profiling.html).
4. Use VTune for microarchitecture + memory bottlenecks (hotspots, uarch-exploration, memory-access), and Advisor for roofline analysis.
5. Use ASan for bounds/UB; use Valgrind when you need slower but deeper leak/use-after-free analysis.

How You Know It Is Actually Training

• train_e2e_latest.json shows loss curve with decreasing trend
• training_loss_curve.json and training_grad_norms.json are populated
• training_parity.json reports PASS for oracle checkpoints (when enabled)
• layout_train_audit.json and memory diagnostics show no bounds/overlap/canary failures
• Generated runtime summary includes expected forward/backward/optimizer kernel calls

Debug Fast: First-Divergence and Memory Safety

What Changes for Backprop vs Inference

Artifact Checklist in a Healthy Run Directory

/tmp/v7_exp1/
  weights.bump
  weights_manifest.json
  ir1_train_forward.json
  ir2_train_backward.json
  ir_train_invariants.json
  layout_train.json
  layout_train_audit.json
  generated_train_runtime_v7.c
  generated_train_runtime_summary_v7.json
  libtrain.so
  train_e2e_latest.json
  training_loss_curve.json
  training_grad_norms.json
  training_parity.json
  memory_diagnostic_latest.json
  ir_report.html

Use this page as the operator baseline: if these artifacts line up and the gates pass, your v7 backprop pipeline is stitched correctly.