Clef-Flash decision-head bring-up

Registered local LoRA jobs

Register an approved package to enable decoder_lora_fp32_v1 through cke.training.jobs.v2. The recipe accepts two-layer dense FP32 BUMP decoders with rms_eps=1e-5, 2–64 fixed tokens, embedding width up to 256 and vocabulary up to 4096. An administrator supplies the package, base files, and tokenizer identity. The trusted local adapter supplies the caller's identity.

Register a package

Save this as package.json alongside the relative directory containing weights.bump and weights_manifest.json. Use actual artifact hashes and label generated weights synthetic. Both splits must contain records and cannot share identical input token sequences. The package is limited to 8 MiB and 4,096 records. Tokenize with the pinned tokenizer before submission.

{
  "schema": "cke.lora.training-package.v1",
  "base": {
    "directory": "base",
    "manifest_sha256": "<64 lowercase hex digits>",
    "weights_sha256": "<64 lowercase hex digits>",
    "provenance": "pretrained"
  },
  "tokenizer": {"id": "<pinned tokenizer ID>", "sha256": "<tokenizer hash>"},
  "tokens": 4,
  "records": [
    {"id": "train-1", "split": "train", "tokens": [0,1,2,3], "targets": [1,2,3,4]},
    {"id": "validation-1", "split": "validation", "tokens": [0,1,2,5], "targets": [1,2,5,4]}
  ]
}

Map an opaque package ID to its local file and SHA-256 in the registry:

{"demo": {"path": "/absolute/package/package.json", "sha256": "<package hash>"}}

Launch the local protocol from a clean CKE checkout with a built engine:

python -m server.training.jobs --lora-registry /absolute/registry.json \
  --store /absolute/private-job-store --principal studio

Prepare, submit and retrieve

  1. Discover executable_recipes, package identities, split sizes and projection targets. Pin the worker boot identity and capability hash.
  2. Prepare with a 32-hex client revision, experiment ID and configuration. Review the returned plan and hash before submitting.
  3. Submit that hash with the same revision and an owner-scoped idempotency key.
  4. Observe progress and terminal status. After a worker restart, an interrupted job reports recovery_required. The configured wall deadline bounds a job.
  5. Retrieve the report, adapter ZIP and checkpoint ZIP by artifact ID. Verify the byte count and SHA-256. Keep the matching base and compiled runtime for LoRASession.load_adapters or load_checkpoint.
{
     "recipe": "decoder_lora_fp32_v1", "package_id": "demo",
     "package_sha256": "<package hash>", "dtype": "fp32",
     "optimizer": "adamw", "method": "lora", "steps": 40,
     "learning_rate": 0.0003,
     "adapters": [
       {"target":"layer.0.wq","rank":2,"alpha":4,"name":"query","seed":41},
       {"target":"layer.1.wv","rank":3,"alpha":6,"name":"value","seed":42}
     ]
   }

Generated C runs forward, loss, backward and AdamW. Successful jobs check frozen weights and exact validation loss and logit hashes after both reloads. Job reports label numerical oracle certification NOT_TESTED and held-out quality MEASURED_NOT_CERTIFIED. This local protocol requires a trusted caller. Remote authentication, cancellation and submitting a checkpoint as a new job are outside the supported contract.

CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q \
  tests/test_v8_lora_jobs.py server/tests/test_training_jobs.py \
  server/tests/test_training_bridge.py server/tests/test_training_contract.py \
  tests/test_v8_lora_session.py tests/test_v8_decoder_lora_training.py
python scripts/nightly_runner.py --tests training_job_lifecycle --require-executed

Bounded notebook LoRA training interface

The public cke.v8.prepare_lora interface accepts an existing FP32 BUMP base artifact and explicit LoRAConfig targets. It resolves a checked plan, freezes all base parameters and prepares externally initialized adapter values before compilation. plan.compile uses the existing shared backward lowering, memory planner and native compiler. No Python gradient calculation, optimizer update or second neural execution schedule is introduced.

plan = cke.v8.prepare_lora(base_directory, tokens=4, adapters=[
    cke.v8.LoRAConfig("layer.0.wq", rank=2, alpha=4, name="query", seed=42),
    cke.v8.LoRAConfig("layer.1.wv", rank=3, alpha=6, name="value", seed=43),
])
with plan.compile(new_run_directory, dataset_id="my-dataset-sha256",
                  max_steps=1000, learning_rate=0.003) as session:
    metrics = session.train_step(token_ids, target_ids)
    validation = session.evaluate(validation_ids, validation_targets)
    session.save_checkpoint(checkpoint_directory)
    session.export_adapters(adapter_directory)

cke.v8.lora_capabilities() reports the supported scope. cke.v8.lora_targets(base_directory, tokens=4) discovers supported projection IDs, layer/role, shapes, legal rank ranges and parameter costs without initializing adapters, compiling or executing training. Initially: two-layer dense qwen3-template FP32 artifacts with explicit RMS/QK epsilon 1e-5, 2–64 tokens fixed at compile time, explicit projection targets, adapter-only native AdamW, and bounded workspace and step count. It does not load arbitrary pretrained Qwen/GGUF models, implement BF16 whole-decoder training or QLoRA, or accept arbitrary PyTorch modules. Token IDs must match the base tokenizer; tokenization is not guessed.

LoRASession.open verifies artifact/build identities and creates isolated native state. Adapter reload requires the same base, shapes, targets and scales and a fresh optimizer. Checkpoint resume instead restores the complete optimizer trajectory and declared dataset/step-budget identity. All checkpoint payloads are validated before import. Calls on one session are serialized.

Tests execute the notebook cells, two concurrent session objects, input rejection and recovery, corrupt checkpoint rejection, incompatible adapter rejection, 40 native updates, and fresh-process reload without importing PyTorch or Transformers. Numerical certification remains in the complete-decoder oracle lane. Training loss is separate from held-out quality and performance.

Examples: version/v8/notebooks/03_generated_lora_session.ipynb and version/v8/examples/python_lora_session_v8.py. Both require an explicit existing base artifact and tokenized records; neither substitutes a synthetic base or silently changes unsupported input. The notebook regression uses a labelled synthetic base solely as test evidence.

make -j4 build/libckernel_engine.so
CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q tests/test_v8_lora_session.py
CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python scripts/nightly_runner.py --tests training_lora_fp32 --require-executed

Complete small decoder with FP32 adapters

attach_lora_adapters(..., freeze_base=True) explicitly freezes all base parameters while preserving activation gradients through the decoder. The default preserves existing selective-training behavior. Initialization remains external; generated native code performs forward, backward and AdamW updates.

The regression uses the standard two-layer dense Qwen-style decoder template, including embeddings, normalization, RoPE, causal attention, residuals, SwiGLU and vocabulary loss. Rank-2 query and rank-3 value adapters in different layers total 320 trainable parameters. Four- and seven-token cases use separate compiled geometries; this does not certify dynamic lengths in one library.

Both cases compare named forward tensors, loss, every adapter gradient, weights and optimizer moments against independent PyTorch for 40 updates. Tests cover frozen-base bytes, zero-B initialization, finite differences, capacity/nonfinite rejection, checkpoint continuation and fresh-process forward-only reload. Missing dependencies remain explicit skips; numerical tolerances are unchanged.

Training loss fell from 4.764195 to 1.113996 and from 3.932966 to 1.229868. Maximum observed gradient error was 7.67e-6, weight error 2.55e-6 and moment error 4.49e-7. This is synthetic training correctness, not held-out application accuracy. Pretrained Qwen fine-tuning, base-import adapter manifests, complete BF16 decoder training, QLoRA and performance certification remain untested.

make -j4 build/libckernel_engine.so
CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q tests/test_v8_decoder_lora_training.py tests/test_v8_lora_training.py tests/test_v8_lora_bf16_base.py tests/test_v8_selective_training.py

The existing training_lora_fp32 nightly lane includes this test. PyTorch runs only as the independent development oracle.

Declared frozen native prefix in training

A checked feature-producing component can feed the existing generated head trainer when its circuit explicitly declares stop_gradient=True. Kernel maps declare a frozen forward boundary and borrowed context arguments; shared lowering owns feature and scratch storage. Generated C executes the component, head forward, backward and AdamW. Python need not transfer hidden features or schedule component tokens.

The caller binds an exclusively owned, live native descriptor through ck_train_set_native_context_ex and retains it until execution ends. Contexts are not parameters or checkpoint contents and must be rebound after initialization or reset. Changed inputs, weight imports, failed execution and evaluation invalidate permission to apply stale gradients. Failed native execution cannot advance the optimizer.

Small non-Clef tests compare five FP32 updates and parameter gradients against PyTorch, including a declared FP32 LoRA adapter after the frozen component. They test invalid IDs, nonfinite output, cancellation and recovery, and run a relocated C-only training host after deleting its build directory. This certifies the bounded interface. A separate released-model run executes the frozen 9B backbone and one generated head update with no Python hidden-feature handoff. Fresh pinned PyTorch head comparisons on the identical native features pass: maximum gradient error 1.02e-7, weight error 2.38e-7 and optimizer-state error 1.01e-8, preserving the existing tolerances. A separate C-only host outside the checkout reproduces the complete updated weight array and loss exactly, without a Python process or captured hidden inputs. Loaded native library identities are retained; this is not arbitrary-machine portability. See bound evidence. This fixed-request experiment is not yet exposed by the application training recipe. No backbone derivative, BF16 adapter or quantized-base training is advertised. Connected backbone/head numerical acceptance remains a separate open gate.

python3 -m unittest tests.test_v8_frozen_component_training -v

BF16 compute in a frozen base, FP32 adapters

The existing projection-precision transformation composes with FP32 LoRA. Attach the adapter first, then select the frozen base operation for use_bf16_projection. Adapter matrices, base parameter storage, gradients and AdamW moments remain FP32. The base projection saves RNE BF16 compute operands and accumulates in FP32; its input gradient still reaches trainable operations upstream.

from version.v8.training.lora import attach_lora
from version.v8.training.bf16_projection import use_bf16_projection

adapted, spec = attach_lora(authored_training_ir,
    target="bridge", name="task", rank=2, alpha=3)
base = next(op for op in adapted["ops"] if any(
    ref["tensor"] == "weight.bridge" for ref in op.get("weights", {}).values()))
mixed = use_bf16_projection(adapted, base["op_id"])
# Prepare checked weights, then use the existing backward lowering/runtime.

The resolved BF16 projection providers are ck_projection_bf16_train_forward and ck_projection_bf16_train_backward in src/kernels/projection_bf16_train.c. They use portable C conversion and ordered FP32 arithmetic, with no BF16 hardware-ISA dispatch; these tests do not claim AVX-512 BF16 or AMX acceleration. Generated provider identities are asserted alongside native execution.

A fourth composition trains rank-two and rank-one adapters across two BF16-compute frozen projections. It requires the shared composition audit alongside independent gradients, unchanged FP32 base/scale bytes and exact fresh-process resume.

Three small generated compositions cover adapter-only training, an upstream trainable projection, and rectangular dimensions. Each compares loss, gradients, weights and optimizer moments for 40 AdamW updates, including checkpoint reload, fresh-process continuation and inference reload. The independent PyTorch oracle implements this declared saved-BF16-operand/FP32-gradient contract; it is not ordinary PyTorch autocast. Frozen matrices and adapter scale remain unchanged. Finite differences are not used to certify a straight-through derivative of rounding. Retained evidence records maximum gradient error 4.10e-7, weight error 1.67e-6 and moment error 7.81e-8 across these trajectories.

This reuses existing providers without a compiler or arithmetic change. It is not BF16 adapter storage, whole-model mixed precision, QLoRA or released Clef adapter certification. Base storage remains FP32 and saved BF16 buffers are additional memory; no speed or memory improvement is claimed without measurement.

python -m pytest -q tests/test_v8_lora_bf16_base.py
python scripts/nightly_runner.py --tests training_projection_bf16 --ci --require-executed

Bounded FP32 adapters and weight preparation

attach_lora_adapters accepts explicit target, rank, alpha and name declarations for multiple unique biasless FP32 projections before backward synthesis. It reuses ordinary projection, scale and add operations and the existing generated training backend. A declared parameter budget is checked against resolved ranks; the authoring utility does not guess targets or reduce ranks silently. Tied/shared targets, unsupported dtypes and inconsistent geometry fail.

Each inserted base projection has an adapter-qualified operation identity and a separate projection semantic label. The original operation identity stays on the join that produces its former output. Name collisions fail before lowering. A two-adapter composition passes the shared provenance audit and independent update comparison.

resolve_lora_parameters returns names, shapes, layout, dtype and trainability without parameter values. prepare_lora_weights validates every base parameter, initializes wholly absent adapters with explicit seeds, and returns schema/specification/weight hashes plus initialization method, implementation and NumPy version. Existing adapters are preserved; partial adapters, stale scale values and unused initialization seeds fail. Resume requires all parameters and performs no initialization. Optimizer moments and counters still resume through the existing checkpoint interface.

from version.v8.training.lora import attach_lora_adapters
from version.v8.training.lora_weights import prepare_lora_weights

adapted, specs = attach_lora_adapters(authored_training_ir, adapters=[
    dict(target="projection.weight", name="task", rank=8, alpha=16),
])
weights, evidence = prepare_lora_weights(
    adapted, base_weights, seeds={"task": 41})
# Compile the adapted graph through the existing training lowering/backend.
# Pass these checked weights to its native loader; Python does not update them.
resumed, resume_evidence = prepare_lora_weights(
    adapted, checkpoint_weights, mode="resume")

This is supported training-IR authoring, not arbitrary PyTorch module conversion or an all-model notebook API. The default initialization remains PCG64 normal A with standard deviation 1/sqrt(input width), zero B and frozen FP32 scale. Initialization is artifact preparation outside native computation. Generated C executes forward, gradients and AdamW updates.

The asset-free LoRA suite compares four nonzero native training compositions with independent PyTorch across 40 updates, including two adapters, rectangular projections, frozen-base gradient flow, evaluation, checkpoint resume and inference export. It also rejects malformed schemas and partial initialization. A shared-builder portability test runs two adapters for 40 updates from a C-only host, moves the bundle, removes the build directory, and reproduces identical results. Frozen base matrices and scales remain unchanged; removing the bundled engine fails. This does not certify BF16 adapters, quantized-base backward, QLoRA or complete connected Clef training.

CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q tests/test_v8_lora_training.py
python scripts/nightly_runner.py --tests training_lora_fp32 --json /tmp/lora.json

Map-owned projection and addition differentiation

FP32 checked row projections (with or without bias) and row addition declare training_contract.differentiation in their kernel maps. The shared circuit adapter resolves the operation interface, ready gradient rule, saved operand ports and dimension aliases from that declaration. A provider name or prefix no longer establishes training support for these operations.

The bounded cke.training.direct-vjp.v1 contract reuses the same forward provider and existing operand tensors. It validates FP32 ports, saved operands and positive bounded dimensions before backward generation. Ready rules must explicitly admit the operation interface and provide exactly the declared input/weight gradient destinations with compatible ownership. Missing, unrelated, duplicate or misowned destinations are rejected even when saved operands match. It does not introduce a new derivative executor or an expression language. Cached-forward rewrites and other operation groups still use explicit translation rules; this is not arbitrary-circuit differentiation or BF16 adapter certification.

Generated nonzero projection compositions, complete supplied-feature head training and mixed-projection regressions retain independent PyTorch checks. Unavailable pinned assets remain SKIP. Connected backbone/head numerical acceptance remains separate and unresolved.

A released-size Ryzen replay compares all 121,762,820 gradient elements and 123 exported tensors exactly with the retained baseline. A relocated C-only host reproduces its update. This is one supplied-feature head update, not a fresh backbone oracle run; the prior connected score FAIL remains visible in the identity-bound evidence.

python -m pytest -q tests/test_v8_map_training_binding.py tests/test_v8_circuit_training_lowering.py tests/test_v8_clef_evaluation.py tests/test_v8_clef_full_training.py tests/test_v8_clef_mixed_training.py

Composition and training audit follow-up

The composition gate now requires the declared consumer operation to contribute to the requested output. A live sibling cannot make a dead declared consumer pass. Boundary identities, shape, dtype and element count must be present and consistent. Structural, executed and numerical verdicts remain separate.

Each head update retains exact frozen-embedding verification through a generated native tensor comparison. It does not export all parameters for this check. Full snapshots remain available for certification and export. On the released 121,762,820-element head, one Ryzen update avoided a 488,296,464-byte host weight snapshot and compared 1,245,184 bytes in place. The native update took 14.153 seconds and its comparison 0.239 ms in that run; these are not whole-job throughput claims.

Broadcast expansion uses declared immutable int32 constants owned by the generated runtime, not a host input named broadcast_zeros. An unrelated broadcast-plus-GELU graph passes 40 PyTorch gradient/AdamW comparisons, including a trainable broadcast row. Maximum gradient, updated-weight and moment errors were approximately 1.27e-7, 1.07e-6 and 4.47e-8 under the existing bounds. Invalid constants and attempts to write them are rejected.

The released-head update and gradients exactly replay the retained independent-oracle baseline; a relocated C-only host also reproduces the update. This is supplied-feature head training, not a fresh connected backbone oracle run. The prior connected score comparison remains FAIL, maximum error 0.009174 at a 2e-6 bound. Evidence and identities.

python -m pytest -q tests/test_v8_generated_composition_contract.py tests/test_v8_circuit_training_lowering.py tests/test_v8_clef_evaluation.py
python scripts/nightly_runner.py --tests generated_composition_training --json /tmp/composition-training.json

Relocatable native head training

The head recipe uses the existing shared training build helper and packages libckernel_engine.so beside train.so, with a $ORIGIN loader path. Build identity records the compiler, generated source, extra provider sources and engine hash. Additional neural sources are resolved from generated provider declarations and kernel maps when their symbols are absent from the engine; the head no longer names a special source file. Missing source contracts fail before compilation. Resume validates the packaged engine rather than consulting the checkout's build directory; older runtimes without that dependency must be regenerated.

A C-only regression host runs generated forward, backward and AdamW after the bundle is moved outside its build directory. The model-neutral graph completes 40 updates with frozen weights unchanged. A complete small Clef head reproduces three native updates and evaluation exactly after relocation. Removing the packaged engine causes startup failure. No Python interpreter or repository headers are required during those deployed runs.

Released-head relocation evidence: On Ryzen, the full 121,762,820-element trainable head completed one AdamW update through a relocated C-only host. Loss changed from 0.0080747874 to 0.0078037283; all updated weights matched the native recipe exactly. Current gradients and updated head tensors also match the retained independent PyTorch-certified baseline exactly. This was a baseline replay, not a fresh oracle run. The single process took 18.22 seconds including initialization and evaluations; it is not a performance benchmark. Artifact identities and separate verdicts retain the connected-model numerical FAIL.

The private full-size reproduction is retained on Ryzen at /data/cke/private/clef-portable-head-20261114/reproduce.py; run it with the pinned oracle environment and a new output directory argument. It consumes the retained native feature capture and released head, so its scope is explicitly head-only. The small asset-free regression remains in ordinary nightly; it does not impersonate this full-size run.

This is runtime portability on the tested Linux ABI, not arbitrary-machine binary portability or released 9B connected-model certification. The test host loads prepared parameters and feature requests; text preprocessing and a declared generated backbone-to-head connection remain separate work. The ordinary recipe's Python layer still manages datasets and reports.

The DSL policy now covers the shared circuit-training adapter, composition gate, checked emitter and bundle compiler, plus shared backward synthesis, training memory planning, code generation and runtime building. Clef and Kokoro are explicitly included in forbidden model-dispatch literals. This checks those policy rules; it is not proof that all compiler behavior originates in IR.

Declared native feature connection: development evidence

A model-neutral native sequence provider resets an exclusively owned decoder, executes its token entry, stages checked FP32 feature rows, resets state, and publishes only after success. Circuit authoring connects those rows to an existing head through ordinary edges. The host supplies inputs and a borrowed component descriptor; it does not run the token loop or copy intermediate feature rows.

The synthetic decoder plus complete small head executes in a relocated C-only process after the build directory is removed. Poisoned hidden features and logits are replaced by native output; missing component libraries fail. Failure and cancellation tests preserve unpublished output and permit recovery. Borrowed descriptors hold exclusive decoder ownership; reentrant encode, close or borrowing is rejected, and failed initialization releases native state.

Released-model execution: one pinned 128-token request now invokes the 9B generated decoder component and complete head through one generated entry, with no Python token loop or captured hidden input. Invalid IDs are rejected and a repeated request succeeds. A separate C-only process outside the checkout reproduces current logits exactly. A repeated C-only run records the actually loaded shared-library hashes and thread environment; neither libpython nor libtorch appears in its loaded-library map. Internal backend dispatch choices remain unobserved. This is a fixed diagnostic artifact, not a persistent arbitrary-text service or connected backward graph.

A fresh pinned PyTorch head comparison on the same current native features produces exactly matching option logits (maximum absolute error 0, unchanged absolute bound 2e-6). This isolates head arithmetic; it does not certify the backbone trajectory.

Numerical limitation: current Python-driven and native token loops agree exactly, but both differ from the older native capture beginning at token index 54. That retained comparison remains FAIL; its cause is unresolved. The independent connected PyTorch failure also remains open. The execution identities and separate verdicts do not convert either discrepancy into a pass. The provider declares no backward support; supplied-feature head training remains a distinct path.

The map declares its call ABI and numerical copy contract. Native-entry parameter types must exactly match the provider declaration; code generation rechecks the binding and rejects altered pointer expressions. Component arithmetic, artifact identity and lifetime remain separate obligations; an exact feature copy does not certify the decoder that produced it.

python -m pytest -q tests/test_v8_native_entry_binding.py tests/test_v8_decoder_feature_sequence.py tests/test_v8_native_feature_head_composition.py
python scripts/nightly_runner.py --tests decision_native_frontend --json /tmp/native-feature-boundary.json

Remaining before deployment: expose the verified composed artifacts through a stable package/loader interface, certify connected outputs, and provide a checked native-context input for frozen component execution in training. The released C-only trial uses existing model artifacts outside the checkout; it does not certify arbitrary-machine portability. Runtime configuration must declare the separate backbone-token input as int32; compact head embedding IDs are not interchangeable with backbone vocabulary IDs.

Model topology lives in the circuit builders here. Kernels implement arithmetic; maps describe ABI, scratch and numerical contracts. Generic operation catalogue entries do not inspect the model name.

Native text → backbone → decision → head training

The offline diagnostic now runs the released Clef-Flash Qwen3.5-9B backbone through ordinary generated decode execution, reads its declared final feature row, and passes those CKE-produced features and pinned output-embedding rows into the generated head. No PyTorch or Transformers import is needed for production inference or head training. Python owns schema encoding, tokenization, artifact preparation, requests and reporting; generated C owns neural forward/backward and AdamW.

Generated composition: NOT_IMPLEMENTED. End-to-end certification: NOT_TESTED. The host currently extracts backbone features, prepares embedding rows and builds a separate head package. Two native components connected by Python do not satisfy the generated backbone-to-head composition gate. This is component integration evidence, not a complete compiled inference/training graph. The CLI option --require-generated-e2e fails before execution on this diagnostic path.

The shared bounded circuit-training adapter now owns operation-to-training translation; Clef authoring supplies its graph, inputs, frozen parameters and grouped loss. Cosine VJP semantics are explicit circuit metadata, never inferred from operation names. The adapter still has a bounded provider inventory; it does not imply arbitrary inference circuits support training. A non-Clef graph exercises generated forward, backward, frozen-weight gradient propagation, evaluation and 40 native AdamW updates against PyTorch. Standard nightly checks reject missing composition evidence and host-fed internal edges. Structural checks remain distinct from execution, gradient and numerical certification.

Feature batches now use the shared generated FP32 input ABI: ck_train_set_float_inputs_ex. Hidden features occupy input storage, not a frozen __features.hidden parameter. Batch binding copies only feature data and does not export/import the parameter array. Counts, alignment, overlap and finite values are checked before publication; rejection invalidates input readiness and leaves parameters and optimizer state unchanged. Backward cannot reuse a forward pass after the feature binding changes. These inputs are nondifferentiable external features; this is not a substitute for a connected backbone activation edge when backbone training is requested.

Checkpoints retain parameters and optimizer state, not feature batches. Inputs must be supplied again after initialization/reload. Existing generated runtimes without this ABI must be regenerated; their old runtime identities are not silently accepted as compatible. This remains a fixed-capacity input contract.

The backbone is frozen. The 121,762,820-element released head is trained using train_clef_head.Runtime, exported and reloaded through generated inference. This does not train the backbone, introduce LoRA, or establish held-out task accuracy. The head is compiled for each offline request geometry; a persistent arbitrary-text application endpoint remains a separate integration gate.

The Qwen3.5 circuit now selects existing BF16-storage providers for recurrent softplus, attention gating and text rotary arithmetic when BF16 recurrent projection weights are declared. Identical-input replay isolated these contract mismatches: FP32 gating differed by 0.00481534, and position-one rotary Q differed by 0.01765776. The existing BF16 providers matched the corresponding PyTorch values exactly. FP32 and quantized selections remain unchanged. The recurrent softplus replay also becomes exact (previous maximum difference 3.55e-9). These are circuit contract fixes, not new model-specific compiler branches.

The diagnostic profile explicitly selects the existing oneDNN 3.12 projection contract. Its reference is Transformers 5.10.2, the release's recorded version, with SDPA attention. An experimental Transformers checkout and eager attention are retained as diagnostic comparisons, not silently treated as the same oracle.

Released-weight evidence, 2026-11-14

On Ryzen 9950X3D, one 128-token sentiment request ran through the actual released backbone and head with PyTorch/Transformers imports blocked. One native AdamW update reduced this record's loss from 0.008074787 to 0.007803728; exported/reloaded logits were exact. This is a training-mechanics test, not a held-out accuracy result.

Independent checkMaximum absolute errorVerdict
Head on identical native features9.53674e-7 (bound 2e-6)PASS
All 121,762,820 head parameter gradients1.22469e-7 (bound 3e-5)PASS
AdamW-updated weights2.38419e-7 (bound 3e-6)PASS
Connected logits versus SDPA full prefill0.00803661 (bound 2e-6)FAIL
Connected logits versus SDPA incremental execution0.00917447 (bound 2e-6)FAIL

The native path executes end to end, but is not numerically certified. The three exact local fixes do not resolve the complete backbone trajectory. The first token's final features are exact; the second token still diverges. Continue identical-input state/operation diagnosis before expanding LoRA. No tolerance was increased and no reference intermediate entered production.

A further state capture identifies a precision-policy difference: the released Transformers incremental cache stores BF16, while CKE's selected recurrent contract retains FP32 state. Diagnostic BF16 rounding of CKE's first-step state reproduces all 524,288 reference values exactly. A separate reference experiment retaining FP32 cache state still has connected logit error 0.00187111 and remains FAIL. This isolates one cause, not all remaining divergence. Production state precision is unchanged; the modified-reference experiment is explicitly labelled diagnostic, not released parity.

Machine-readable evidence retains raw logits, RMSE, reference signal magnitudes, identities, dependency hashes, training results and local replay results. Private activation packets remain on the asset host; the public artifact contains the synthetic request and summary metrics.

Reproduction and boundaries

Use the pinned release fb4417d31c1d0f9e7e77f1141b40aaede7491e00, its SHA-verified release lock, and standalone native oneDNN/SLEEF/MKL dependencies. This profile is validated on AVX-512-capable Ryzen; it is not an AVX2-only portability claim. Set CK_ONEDNN_LIBRARY, CK_SLEEF_LIBRARY and CK_MKL_LIBRARY to those installed libraries. Record their hashes and thread settings. The build command requires a fresh output directory; it never reuses unbound converted weights. A local trusted administrator must supply the expected lock and bundle hashes.

python3 version/v8/decision/build_native_clef_backbone.py \
  --release "$RELEASE" --lock "$LOCK" --lock-sha256 "$LOCK_SHA" --output "$BUNDLE"
python3 version/v8/decision/run_native_clef.py \
  --release "$RELEASE" --lock "$LOCK" --lock-sha256 "$LOCK_SHA" \
  --bundle "$BUNDLE" --bundle-descriptor "$BUNDLE/bundle.json" \
  --bundle-sha256 "$BUNDLE_SHA" --request request.json --output native-run
python3 version/v8/decision/train_native_clef.py \
  --release "$RELEASE" --native-run native-run --report-sha256 "$REPORT_SHA" \
  --labels labels.json --output training-run --steps 1 --workspace-mib 4096
python3 tests/run_clef_native_oracle.py \
  --release "$RELEASE" --native-run native-run --native-sha256 "$REPORT_SHA" \
  --capture "$CAPTURE" --capture-sha256 "$CAPTURE_SHA" --output comparison.json
python3 tests/run_clef_native_training_oracle.py \
  --release "$RELEASE" --native-run native-run --native-sha256 "$REPORT_SHA" \
  --training training-run --training-sha256 "$TRAINING_SHA" \
  --labels labels.json --output gradient-comparison.json

The independent capture tool accepts --attention sdpa --schedule incremental to match the native token-by-token schedule. Its defaults remain eager/prefill for existing users. Full-prefill and incremental comparisons are retained separately; changing the oracle schedule is not permission to erase a failing comparison. Neither reference features nor reference logits enter the native runner. Reports separate execution, same-input head parity, connected logit parity, exact backbone-storage agreement, gradients, optimizer updates and reload. Unavailable reference evidence is NOT_TESTED, not PASS.

The released geometry needs approximately 2.51 GB of native training workspace. The offline CLI explicitly permits a 4 GiB budget; existing application recipes retain their 2 GiB default. This is an arena bound, not total process memory: weights, compiler, Python arrays and exported snapshots require additional RAM. Evaluation performs no optimizer update. Offline tools do not claim HTTP deadlines, cancellation, concurrent backbone ownership, streaming training datasets or automatic application admission.

Cheap standard-nightly coverage checks schema spans, rejection without truncation, BF16 versus FP32/quantized contract selection, artifact tampering, decoder ownership, short/long/short state reset, output lifetime, partial failure and recovery, and a nonzero generated head against an independent reference. Full 9B/released-head runs require the pinned assets and are reported separately. Set CKE_CLEF_NATIVE_ACCEPTANCE to a trusted JSON file with the runner arguments release, lock, lock_sha, bundle, bundle_descriptor, bundle_sha, request and a labels mapping (question ID to candidate ID) to enable the released asset execution/training/reload test. Without that configuration it reports SKIP. That test is execution evidence; run the independent comparison commands separately.

FP32 LoRA: bounded graph authoring and native training

version/v8/training/lora.py now attaches one explicitly named adapter at the existing authored training-IR boundary. This is not automatic adapter admission for arbitrary pretrained models or a new training service endpoint. The selected projection must already belong to a differentiable, bounded, contiguous FP32 training graph. Tied targets, bias, padded strides, BF16 and quantized bases are rejected.

from version.v8.training.lora import attach_lora, initialize_lora
adapted_ir, spec = attach_lora(
    training_ir, target="bridge", rank=2, alpha=3, name="demo")
initial_adapter_weights = initialize_lora(spec, seed=41)
# Merge these new weights with the original weight values, then use the
# existing backward synthesis, layout planner, code generator and AdamW.

For row-major inputs, the declared graph computes X Wᵀ + (alpha/rank) (X Aᵀ) Bᵀ. The base weight is frozen; other parameter declarations are preserved. A uses seeded PCG64 normal initialization, B starts at zero, and the scale is an immutable training parameter. This leaves the initial projection unchanged. Parameter count is rank × (inputs + outputs). The descriptor is retained in authored IR; generated runtime identity and existing checkpoint contracts remain in force. Dedicated portable adapter packages and application-job admission remain follow-up work.

No new kernel, map, or generic compiler branch is introduced. Checked projection, vector scale and add maps supply forward and backward contracts; the shared trainer accumulates both activation-gradient branches and updates only trainable weights. Frozen parameters receive no optimizer state. This is correctness bring-up, not a claim that all frozen-weight arithmetic is optimally pruned.

Independent PyTorch FP32 comparisonNative loss, 40 AdamW updates
Only A/B trainable; all original weights frozen0.73031437 → 0.14123474
Square target, trainable upstream/downstream projections1.42433834 → 0.00277238
Rectangular target (6 inputs, 3 outputs)1.40610135 → 0.00002841

Across these cases, maximum absolute gradient error was 4.48e-7 or less; weight error was 1.20e-6 or less. Existing numerical tolerances were unchanged. Tests compare every trainable gradient, weight and optimizer moment on every update, verify frozen weights bit-for-bit, check a central finite difference, resume at step 20 in both the same and a fresh process, and reload final weights for generated inference. With identity feature inputs, the upstream projection gradient is the transpose of the adapter input gradient, testing both derivative branches. A starts with zero gradient and acquires a nonzero gradient after B's first update.

make -j4 build/libckernel_engine.so
python -m unittest tests.test_v8_lora_training -v
python scripts/nightly_runner.py --tests training_lora_fp32 --require-executed

Use a Python environment with NumPy and PyTorch for the independent oracle. Missing dependencies or the native engine produce explicit SKIP; the required lane rejects SKIP. Retained local evidence. Full-model LoRA, BF16 adapters, QLoRA, Muon and application training-job admission are NOT_TESTED / unsupported by this milestone. Existing Clef FP32 and mixed-BF16 verdicts are not changed by these small adapter tests.

Validation loss correction: forward plus declared loss, without an update

A real notebook consumer at aa68f5812 exposed stale held-out metrics. Runtime.evaluate() ran forward and read ck_train_get_loss(), but that accessor returns cached loss. The grouped loss seed executes at the start of backward, so forward alone returned zero after initialization and the last training loss after updates. Native training, exported weights and scoring were correct; the reported validation metrics were not.

The generic generated ck_train_eval_step(float *loss_out) now executes forward plus the single declared loss-seed operation. It does not run parameter backward or an optimizer step. The existing loss provider refreshes its output-gradient seed and scratch as well as the scalar loss; forward activations and supplied feature inputs also change. Trainable weights, parameter gradients, AdamW moments and update/ accumulation counters remain unchanged. A failed evaluation does not publish a scalar. Ordinary decoder cross-entropy and grouped question loss use the same contract; there is no Clef-specific compiler branch or Python loss calculation in production.

40-update notebook reproductionOld reportCorrected native reportIndependent grouped CE
Validation before training0.00.22789864240.2278986458
Validation after training0.11846388880.17800277470.1780027814

The reproduction uses the original pinned synthetic package, 20 epochs, learning rate 0.001 and 40 native updates, with PyTorch/Transformers imports blocked in the production process. All 40 training losses, trained weights, optimizer-state hash and record logits are unchanged. The original notebook evidence is preserved. Retained numerical evidence and artifact identities separates the old and corrected reports.

Regressions cover initial evaluation, different records and labels, unequal option counts, validation between updates, rejected evaluation and recovery, independent FP64 stable log-sum-exp, multiple held-out records, checkpoint reload, final report metrics and published job metrics. They separately preserve inference export/reload checks. Validation-before means the state at the start of this invocation; when resuming, it is evaluated at the loaded checkpoint, not at the original initialization.

The generated runtime contract now includes evaluation semantics. Regenerate old training libraries; a cached runtime without the new entry rejects rather than silently using stale loss. Existing reports must be regenerated, not relabeled. Numerical bounds are unchanged, and this fix does not turn the mixed-BF16 trajectory into a pass or establish task quality.

python -m unittest tests.test_v8_clef_evaluation tests.test_v8_grouped_loss_training tests.test_v8_selective_training -v
python scripts/nightly_runner.py --tests training_clef_registered_recipe --require-executed

Training bridge v2: opaque revisions and pinned worker identity

cke.training.jobs.v2 adds a bounded local contract for external application backends. It preserves the application's 32-character lowercase hexadecimal client_revision verbatim in the reviewed plan and durable job. It does not cast or hash that revision into an integer. Existing v1 integer-revision clients continue to use their unchanged contract. V2 plans/jobs carry a null legacy draft_revision and explicitly versioned schemas.

This milestone is trusted local integration only. It opens no socket, adds no remote authentication, and does not reuse read-only control discovery for training execution. The principal comes from the trusted supervisor, never JSON. A future authenticated transport must bind a verified backend credential to the appropriate tenant/owner; a caller-selected owner or a tunnel is not authorization.

Lifecycle and freshness

  1. Discover approved recipes and registered package IDs/hashes, geometry and split counts. Retain the returned worker_boot_id and capabilities_sha256. The latter hashes the complete v2 capability body, excluding its own digest. Hashes detect changed content; they do not authenticate a peer.
  2. Every non-discovery request includes both identities and the exact client revision. Preparation validates the configuration and persists a reviewable plan without executing training. The plan hash binds owner, configuration, revision, backend capabilities and expiration.
  3. Submit the reviewed plan_sha256 with a stable owner-scoped idempotency_key. An uncertain timeout must retry the same plan/key/revision; it must not invent a new key. An exact duplicate returns the existing receipt.
  4. Observe and retrieve artifacts only for the authenticated supervisor's owner and matching revision. A different application draft must not publish an older job as current. The independent client rejects mismatched request hashes, boot/capability identities, revisions and corrupted artifact streams.
  5. After worker restart, discover its new boot identity before observing or retrying. An existing idempotency receipt can be recovered, but ambiguous interrupted jobs stay recovery_required; they are never automatically relaunched.

Revisions are opaque, not ordered counters. CKE binds them to the submitted plan; the application remains responsible for tracking which draft is current. Unknown or unavailable capabilities remain unavailable. Missing metrics are absent, never fabricated as zero; clients should display them as unavailable. Cancellation remains unsupported.

Local setup and wire examples

Use the existing administrator package registry and private job store described below. The principal argument is supervisor configuration, not a field accepted from clients. The following command performs discovery only and contains no credentials:

printf '%s\n' '{"schema":"cke.training.jobs.v2","request_id":"discover-1","operation":"discover"}' | \
  python -m server.training.jobs --store /private/jobs --principal demo \
  --clef-registry /private/registry.json

A successful reply uses these fields (placeholders below are explanatory, not a valid pinned response):

{"schema":"cke.training.jobs.v2","request_id":"discover-1",
 "request_sha256":"SHA256_OF_EXACT_REQUEST","worker_boot_id":"BOOT_HEX_32",
 "capabilities_sha256":"CAPABILITIES_HEX_64","ok":true,"error":null,
 "result":{"schema":"cke.training.jobs.v2","worker_boot_id":"BOOT_HEX_32",
   "capabilities_sha256":"CAPABILITIES_HEX_64",
   "operations":["discover","prepare","submit","observe","artifact"],
   "cancel":"unsupported",
   "transport":"trusted_local_only; remote authentication NOT_IMPLEMENTED"}}

The actual discovery result additionally contains the complete registered validation capabilities, execution scope, limits and authorization description. Never hash or pin this abbreviated example. Prepare with the actual returned identities:

{"schema":"cke.training.jobs.v2","request_id":"prepare-1","operation":"prepare",
 "worker_boot_id":"BOOT_HEX_32","capabilities_sha256":"CAPABILITIES_HEX_64",
 "client_revision":"00ffffffffffffffffffffffffffffff","experiment_id":"decision-demo",
 "config":{"recipe":"clef_head_fp32_v1","package_id":"demo",
   "package_sha256":"REGISTERED_PACKAGE_SHA256","dtype":"fp32","optimizer":"adamw",
   "method":"head_only","epochs":2,"learning_rate":0.001}}

Submit requires plan_sha256 and idempotency_key; observe requires job_id; artifact requires job_id, artifact_id and offset. All include schema, request ID, operation, boot/capability identities and client revision. Unknown fields, client paths, principals and shell commands reject. Artifacts retain the existing 8192-byte chunk limit and manifest SHA-256 verification; an application must retain the expected manifest and verify the entire resumed transfer.

Evidence and remaining agreement

Asset-free independent consumer tests cover no-execution preparation, exact revision preservation, duplicate/lost-reply recovery, restart, stale identities, cross-owner denial, wrong revisions and corrupt/truncated artifact responses. These tests simulate execution; they do not certify native arithmetic or the actual laptop application. The separate opt-in native Clef job test exercises opaque revisions through training and export.

Before exposing a remote service, agree and test authenticated ZeroMQ peer/tenant mapping, server identity pinning, bounded messages and deadlines, disconnect/reconnect behavior and transport error envelopes. V2 local error handling remains the existing stdio rejection envelope. Do not advertise a production network endpoint or cancellation until those contracts exist. No backward kernels, optimizer math or compiler lowering change in this bridge milestone.

python -m unittest server.tests.test_training_bridge server.tests.test_training_jobs -v
python scripts/nightly_runner.py --tests training_job_lifecycle --require-executed
CKE_RUN_CLEF_JOB=1 python -m unittest \
  tests.test_v8_clef_training_recipe.NativeRecipe.test_opaque_revision_native_job_and_export -v

Application training: larger supplied-feature packages

The FP32 head recipe now admits up to 4,096 records and 512 token positions, with 1–1,000 epochs and a combined ceiling of 100,000 updates. The loader retains its 512 MiB decoded-dataset bound and the compiler retains its 2 GiB native-workspace bound. These are admission ceilings, not evidence that every maximum-size geometry fits memory or has acceptable speed. Package manifests are bounded to 8 MiB.

Application clients must rediscover capabilities: clef_head_fp32_v1.limits now explicitly reports records, tokens, update budget, dataset bytes and the exact-length restriction. The independent simulated consumer and job-progress checks use the expanded contract. An administrator can construct Jobs(..., wall_seconds=3600) for a longer local run; the default remains 600 seconds and the accepted range is 60–86,400 seconds. This is local policy, not a client-requested timeout override. Cancellation and remote authentication remain unsupported.

The trainer indexes the schedule instead of materializing one record reference for every update. Every 1,000 updates before the final/interrupted step it evaluates validation records and publishes an identity-bound checkpoint. Only the last two periodic checkpoints created by that invocation are retained; a resumed invocation preserves earlier files. The final checkpoint and explicit stop-after behavior remain available. Resumption requires the same dataset and configuration. Older checkpoint configurations without the new checkpoint/evaluation policy are not silently accepted.

The final report includes per-split, per-question accuracy, macro-F1, support and confusion matrices, plus record logits, labels and stable per-question softmax scores. Candidate identities, not their array positions, define classes. Question IDs must denote the same task across records. Macro-F1 includes declared classes with zero support (zero-division score zero); softmax scores are not calibrated confidence. Sparse confusion entries are always supplied; dense matrices are omitted once their combined 65,536-cell budget is exhausted. This avoids quadratic reporting memory for many candidate identities. Reports include elapsed time and process high-water RSS on Linux. Timing includes preparation/export and is not a kernel-throughput measurement.

Serving fixtures no longer grow with the training dataset: export registers at most 64 question examples, prioritizing the first validation record then dataset order, and explicitly reports inference_export_record_ids. Native training-runtime evaluation still reports all records. Independent generated-inference export checks apply to the selected subset; the exported head contains the trained weights, not a claim of corpus-wide export parity.

Executed stress test: 34 synthetic records, 96 token positions, 41 epochs / 1,353 updates, periodic validation at update 1,000, and 32 exported two-question records. This tests limits and lifecycle behavior, not sentiment accuracy or Qwen execution. The existing distinct-record pinned PyTorch head-update comparison also passes without tolerance changes.

make -j4 build/libckernel_engine.so
python -m unittest tests.test_v8_clef_dataset_scale -v
python scripts/nightly_runner.py --tests training_clef_registered_recipe --require-executed
python version/v8/decision/train_clef_head.py \
  --package /absolute/path/package.json --package-sha256 EXPECTED_SHA256 \
  --run-dir /absolute/path/new-run --json-out /absolute/path/result.json \
  --epochs 50 --lr 0.001

Still required before claiming general real-text training: each package has one exact token/question/option geometry. No implicit padding or attention masking is provided. Mixed-length capture, bucketed shared-optimizer training, a checked multi-record capture-to-package converter, automatic best-validation-checkpoint selection, and a separate final test-set workflow remain follow-ups. The application should preserve train/validation/test separation and must not select models using the final test set. Native backbone execution, full-model LoRA and QLoRA are not enabled by this expansion. No real dataset benchmark or application-quality claim has been made.

Registered FP32 Clef head training

This opt-in recipe trains the complete supported Clef head on supplied backbone features. It does not execute or train a backbone, tokenize text, add adapters, or execute application actions. A successful job means generated training, frozen embedding checks, checkpoint reload and generated inference export passed. It does not certify the submitted dataset's quality, calibrated confidence or numerical parity against an oracle. Those report fields remain NOT_TESTED.

Ownership and precision

The existing v8 circuit authoring, backward rules and shared training code generator produce native forward, backward and AdamW execution. Python validates records, loads input features, invokes complete generated steps and packages artifacts. It never computes model gradients or updates parameters. No new compiler branch, model scheduler or training stack is introduced.

The admitted recipe is clef_head_fp32_v1: FP32, AdamW, per-question cross entropy, 1–1,000 epochs (at most 100,000 updates), learning rate (0, 0.01], epsilon 1e-4, betas (0.9, 0.999), weight decay 0.01, no dropout, gradient accumulation 1. All head parameters train; output embedding rows and supplied backbone features are frozen. Mixed BF16, LoRA and QLoRA are unavailable through this recipe. In particular, the existing mixed-BF16 trajectory failure has not become a pass.

Dataset/package contract

cke.clef.training-package.v1 has exactly five fields:

  • schema: the version above.
  • model: id, revision, backbone_id, backbone_revision, tokenizer_revision, and feature_provenance: "supplied". Revisions are pinned hexadecimal identities. Supplied provenance records a declaration, not proof that CKE executed that backbone.
  • geometry: tokens, channels, width, heads, feedforward, vocabulary, questions, options. All records share this declared geometry.
  • weights: local basename file and sha256 of the FP32 NPZ raw head weights, including output_embedding.weight and packed QKV weights.
  • records: 2–4,096 records with id (at most 60 characters), split (train or validation), asset (file, sha256), question_ids and grouped candidate_ids. Both splits must be present. IDs preserve question/option order.

Each record NPZ contains hidden (FP32) and int32 input_ids, question_spans, option_spans, global_span, option_counts, question_type_ids, labels. Spans are half-open. Global span must select the last valid token. Labels index that question's allowed options; option counts partition flattened option rows. Arrays, finite values, IDs, counts, shapes, archive expansion and all asset hashes are checked before code generation. Archives must not contain Python objects. See version/v8/decision/training_package.py for exact bounds and inventories. The tests create synthetic, nonzero examples with overlapping spans and two questions with unequal option counts. Do not use private customer data for the first client demonstration.

Run locally

Build build/libckernel_engine.so first. Use a trusted package and retain its hash:

python version/v8/decision/train_clef_head.py \
  --package /absolute/path/package.json --package-sha256 SHA256 \
  --run-dir /private/new-run --json-out /private/result.json --epochs 2

The output contains native runtime/source identities, held-out loss before/after, per-record raw logits and identities, an identity-bound optimizer checkpoint, trained_head.npz, and serving.zip. The latter contains the ordinary generated head package and registered supplied feature artifacts. The recipe loads it with NativeHead and compares its logits with the trained runtime for every record.

For a deliberate CLI checkpoint, add --stop-after 2; continue with the same package, configuration and run directory using --resume /private/new-run/checkpoint-2. This restores weights, AdamW state and step count. The saved runtime and engine identities must match. Completed checkpoints cannot be resumed as new training. This is local CLI resume, not a job-service resume or cancellation promise.

Independent client and job service

The existing cke.training.v1 discovery/validation API remains non-executing. An administrator can opt into the head recipe by registering a package mapping:

{"demo":{"path":"/absolute/path/package.json","sha256":"SHA256"}}

Launch trusted local stdio, from a clean committed source tree:

python -m server.training.jobs --store /private/jobs --principal demo \
  --clef-registry /private/registry.json

Use discovery before constructing a request. A head configuration is:

{"recipe":"clef_head_fp32_v1","package_id":"demo","package_sha256":"SHA256",
 "dtype":"fp32","optimizer":"adamw","method":"head_only",
 "epochs":2,"learning_rate":0.001}

prepare binds configuration, worker boot, capability/source/interpreter/package identities and expiration. submit returns a durable acknowledgement with an idempotency key. Only one job executes, with no backlog; discovery and preparation do not execute training. observe reports durable monotonic progress and metrics. Head metrics use step_kind: update and unit: nats/question; they must not be interpreted as decoder token losses. Artifact retrieval uses opaque IDs, bounded chunks and hashes. The independent clients in server/training/consumer.py and job_consumer.py do not import the worker implementation.

Readiness and gaps

Boundary Status
Supplied-feature FP32 generated head training and inference export Implemented; synthetic native regression
Independent small-head PyTorch comparison Optional pinned-source test; explicit skip when unavailable
Frozen output embeddings and fresh-process CLI resume Native regression
Held-out loss Reported; no automatic quality acceptance
Backbone execution, text-to-decision Not provided by this recipe
Job cancellation / automatic restart / remote resume Unsupported; interrupted jobs become recovery_required
Remote authenticated transport Unsupported; trusted local stdio only
BF16 trajectory certification, LoRA, QLoRA Not admitted

Do not expose this interface as an authenticated remote service. A caller-provided principal is meaningful only inside a trusted local supervisor. A tunnel is a development bridge, not authorization. Evaluation continues to use the separate cke.decision.evaluate.v1 contract; neither evaluation nor control discovery is a training command. Keep the current decision HTTP worker loopback-only.

The service exports checkpoint metadata and serving artifacts, but remote checkpoint-state transfer/resume is not yet an application contract. Applications must surface recovery_required, NOT_TESTED and unsupported operations explicitly.

Validation retained for this change

On the local Intel i7-14700T host with PyTorch 2.12.1+cpu and one requested OpenMP thread:

  • Focused recipe and existing service/client suites: 31 passed, 2 explicit skips (opt-in real SVG and Clef jobs).
  • Real registered Clef job rerun separately from the clean committed tree: passed, including independent observation and chunked artifact hash verification.
  • training_clef_registered_recipe through the standard nightly runner: passed.
  • Existing independent FP32 complete-head 40-update trajectory: passed. Loss 0.28329045 → 0.06633431; max absolute gradient error 1.374632e-6, weight error 1.668930e-6, optimizer-moment error 3.147870e-7. Checkpoint/resume and generated inference export passed. These features and head geometry are synthetic; this run does not certify released-geometry training or task quality.

Reproduce the new boundary with:

OMP_NUM_THREADS=1 CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py \
  python -m unittest tests.test_v8_clef_training_recipe \
  server.tests.test_training_contract server.tests.test_training_jobs -v
OMP_NUM_THREADS=1 CKE_RUN_CLEF_JOB=1 python -m unittest \
  tests.test_v8_clef_training_recipe.NativeRecipe.test_registered_job_and_independent_artifact_client -v
python scripts/nightly_runner.py --tests training_clef_registered_recipe --require-executed

The oracle source must match the SHA-256 in tests/fixtures/decision/clef_flash/identity.json; no source means explicit skip. The production resume test blocks imports of torch and transformers.

FP32 memory-projection boundary: useful diagnosis, still not certified

The bounded --mixed --fp32-memory-projection experiment retains the existing FP32 memory projection in the authored training graph. Other selected projections retain BF16 operands. The independent PyTorch reference uses its original FP32 linear operation for that same projection. The report lists actual selected providers and the exception. There is no production kernel, map, compiler or application admission change.

All five experiments were rerun on the same build of main f6f5e6b1e, seed 611, CPU PyTorch 2.12.1. The complete small head still uses synthetic frozen features and a bounded task; these are not generalization or performance measurements.

40-update comparisonWorst gradient errorVerdict
FP32 baseline1.375e-6PASS
All selected projections mixed, same state6.748e-4FAIL at update 18
FP32 memory exception, same state4.170e-5FAIL at update 37
All selected projections mixed, independent trajectory0.0328365FAIL from update 8
FP32 memory exception, independent trajectory0.00409246FAIL, first at update 21

The exception removes the earlier memory-projection rounding failure, but a new same-state rounding boundary appears at evidence1_feedforward_norm_op → evidence1_feedforward1. Its first affected input is -1.4179686307907104 versus -1.41796875, rounding to -1.4140625 versus -1.421875. Identical-input projection replay gives zero output error and dW error 5.821e-11. The independent trajectory additionally encounters a QKV master-weight rounding boundary. This is evidence against treating one FP32 exception as a general solution.

Thresholds are unchanged. Both mixed policies retain numerical FAIL. Native state hashes match across same-state/independent diagnostic modes for each policy; resume remains exact. Released-weight mixed precision, mixed inference export and performance remain NOT_TESTED. Full reports and packets are referenced in docs/notes/clef_fp32_memory_boundary_20261113.json.

python tests/run_clef_training_oracle.py --mixed --fp32-memory-projection --same-state \
  --report /tmp/clef-boundary/same-state.json
python tests/run_clef_training_oracle.py --mixed --fp32-memory-projection \
  --report /tmp/clef-boundary/independent.json

Application-agent readiness

The numerical experiment above is separate from the registered FP32 head-training recipe. With an administrator-registered supplied-feature package, the local job service now admits clef_head_fp32_v1, executes generated forward/backward and AdamW, and exports a native inference package. Clients must discover the running worker's capabilities before submitting; clef_head_v8 is not a supported alias.

This is head-only integration, not raw-text/backbone execution. BF16 certification, LoRA, QLoRA, remote authentication and job cancellation remain unavailable. The mixed-policy numerical failures above remain unchanged.

Same-state update diagnosis: 39 of 40 comparisons pass

--same-state copies CKE's pre-step master weights, both AdamW moments and optimizer step into PyTorch before each diagnostic comparison. Packed Q/K/V ownership and scalar parameters are restored explicitly and checked by byte-exact round trips through the effective-weight mapping. No oracle value is copied into CKE. The independent trajectory remains a separate required experiment.

For the same small complete mixed head, only zero-based update 18 fails unchanged numerical gates. Maximum gradient, master-weight and moment errors are 6.748e-4, 2.578e-5 and 6.748e-5. This is still numerical FAIL, not complete mixed-precision certification. The earlier independent update-8 failure disappears when master weights and optimizer state are synchronized.

The remaining first significant divergence is hidden_norm → memory_projection. With identical inputs and normalization parameters, native and PyTorch LayerNorm differ by at most 2.384e-7. One value is 1.0976563692092896 versus 1.09765625; these round to BF16 values 1.1015625 versus 1.09375. Replaying the projection on identical operands gives zero output error, dW error 1.863e-9 and isolated dX error 4.657e-10. The packet retains the LayerNorm operands and the rounding witness. This exposes amplification at a precision boundary; it does not justify changing global tolerances.

Input gradients may contain contributions from multiple branches. The diagnostic now replays the selected provider's dX into its own buffers rather than comparing it with the accumulated input gradient. A regression deliberately omits the accumulated dX to enforce this distinction.

CKE's forty native losses, final master-weight hash and final optimizer-state hash are identical in same-state and independent runs. Checkpoint/restart remains exact. The report labels same-state comparison explicitly and marks independent trajectory parity NOT_TESTED in that report; the separate independent run remains FAIL. Production kernels, maps and compiler behavior are unchanged by this diagnostic.

python tests/run_clef_training_oracle.py --mixed --same-state \
  --report /tmp/clef-oracle/same-state.json
# Keep the independent run alongside it; both currently exit 1.
python tests/run_clef_training_oracle.py --mixed \
  --report /tmp/clef-oracle/independent.json

Evidence: docs/notes/clef_same_state_oracle_20261113.json. Next: evaluate the declared LayerNorm-to-BF16 precision boundary and a justified mixed-precision acceptance policy, preserving the independent FP32 and mixed trajectory checks. Do not force backend rounding history merely to obtain PASS.

Complete-head PyTorch comparison: FP32 passes; mixed trajectory fails

The complete small topology now runs forty independent native/PyTorch AdamW updates with RNE BF16 projection operands and FP32 masters, bias, gradients, optimizer state and remaining operations. Biased projections use explicit generic forward/backward maps. Backward synthesis still owns the derivative graph; there is no Clef-specific compiler dispatch or Python update of CKE weights.

This is diagnostic mixed-precision training, not mixed-precision certification. The pinned reference owns the head topology; a test-only PyTorch autograd adapter declares the projection precision and identity-rounding VJP. This is not default autocast. Inputs are identical synthetic frozen backbone features, with two questions and unequal option counts. It is not complete backbone training.

ComparisonMeasured result
FP32 complete small head, 40 updatesPASS; loss 0.28329045 to 0.06633431; maximum gradient error 1.375e-6.
Released FP32 head, 4 updatesPASS; pinned weights, synthetic frozen features; loss 0.59229440 to 0.13316446.
Mixed small-head trajectory, 40 updatesFAIL under unchanged bounds; native loss 0.28323126 to 0.06657010. Maximum gradient, weight and moment errors: 0.0328365, 0.00121975 and 0.00366078.
Mixed same/fresh-process resumePASS, exact continuation. Separate generated mixed inference export: NOT_TESTED.

At zero-based update 8, the first significant projection divergence is fields0_cross_out. Weight [7,2] is 0.0720214918255806 in CKE and 0.07202146202325821 in PyTorch. Those nearby FP32 values round to different BF16 values: 0.072265625 and 0.07177734375. Replaying the CKE operands in PyTorch yields zero output, weight-gradient and bias-gradient differences at this boundary; input-gradient error is 1.164e-10. This identifies a rounding-boundary amplification of prior master-weight drift, not a demonstrated local projection derivative defect. It does not establish that all later discrepancies are acceptable.

The oracle runner compares raw option logits, loss, every parameter gradient, weights and optimizer moments throughout the trajectory. It retains per-parameter RMS/magnitude metrics, failed gates, artifact hashes and a first-divergence replay packet. Numerical FAIL exits 1; missing assets exit 2. Execution, loss reduction and exact resume cannot override numerical FAIL. Existing bounds are unchanged.

make build/libckernel_engine.so
export CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py
python tests/run_clef_training_oracle.py --report /tmp/clef-oracle/fp32.json
CKE_CLEF_FLASH_HEAD_PATH=/path/to/pinned/joint_head.safetensors \
  python tests/run_clef_training_oracle.py --released --report /tmp/clef-oracle/released.json
# Expected exit 1: complete diagnostic comparison, known numerical FAIL.
python tests/run_clef_training_oracle.py --mixed --report /tmp/clef-oracle/mixed.json
python -m unittest tests.test_v8_clef_mixed_training -v

Recorded measurements and artifact identities: docs/notes/clef_full_training_oracle_20261113.json (Intel Core i7-14700T, CPU PyTorch 2.12.1; no Spark/GPU or performance comparison).

Standard nightly adds biased-provider rejection/autograd, full-topology backward-generation and report-verdict regressions to the existing projection suite. That suite does not certify the failing mixed trajectory. Next: distinguish local same-state update accuracy from independent BF16 trajectories before proposing any mixed-precision acceptance policy. Released mixed training, mixed inference export, LoRA, quantized training, hardware speed and generalization remain unproven.

Bounded BF16 projection training with FP32 masters

One mixed-storage projection graph is certified; complete BF16 Clef training remains unsupported. The authoring helper version/v8/training/bf16_projection.py replaces an explicitly selected projection with a map-declared provider. Ordinary backward synthesis creates the VJP and accumulation edges; the existing generated trainer performs all forward, backward and AdamW arithmetic. There is no model-name compiler branch.

The precision policy is explicit: FP32 inputs and master weights are rounded at each forward projection boundary to BF16 using round-to-nearest, ties-to-even. Those saved compute operands occupy typed uint16_t buffers. The provider computes ascending-index FP32 products and sums without FMA contraction and publishes FP32 output. Backward consumes FP32 output gradients and the saved BF16 operands, producing FP32 input and master-weight gradients. Operand rounding uses a declared identity derivative; this is not the derivative of a discontinuous quantizer, nor a claim of equivalence to default PyTorch autocast. The independent reference uses explicit BF16 rounding and FP32 matmul/autograd with the same chosen VJP. It does not call the CKE kernels.

Master weights, gradient accumulation and AdamW moments remain FP32. Frozen projections still propagate activation gradients, but have no parameter-gradient publication or optimizer state. Compute operands refresh on every forward call after updates or checkpoint restore; derived caches are not checkpoint payloads. The runtime admits BF16 only for nonpersistent, nondifferentiable auxiliary tensors with a typed forward producer and typed map consumers. BF16 model weights, BF16 gradient buffers and a global BF16 training configuration still fail explicitly. Layout byte extents and pointer types are checked; canaries reserve whole 32-bit words around packed caches, with direct kernel tests checking the exact 16-bit tail.

Generated experimentEvidence
4-channel projection and frozen bridge40 native updates: loss 1.42442214 to 0.03998315.
5-to-7 projection, frozen 7-channel bridge40 native updates: loss 1.37150788 to 0.00395414; odd BF16 cache extents.
Independent trajectory comparisonsWorst absolute gradient error 5.41e-8, master-weight error 9.54e-7, moment error 2.43e-8 (PyTorch 2.12.1 CPU, seed 837).
StateByte-unchanged frozen weights, exact same-process and fresh-process checkpoint continuation, trained-weight forward reload.
Primitive contractsRectangular/tail shapes, O0/O3 equality, FP32 dX/dW oracle, nonfinite and finite-overflow rejection, null/alignment/overlap/capacity rejection, unchanged published outputs and recovery.
make build/libckernel_engine.so
python -m unittest tests.test_projection_bf16_train_kernel tests.test_v8_bf16_projection_training -v
python scripts/nightly_runner.py --tests training_projection_bf16 training_selective_parameters --ci --require-executed

Missing PyTorch or the built native engine reports SKIP, not parity PASS. This scalar implementation establishes semantics, not acceleration or an end-to-end memory reduction. It retains FP32 external activations in addition to BF16 saved compute operands. Full-head mixed precision, BF16 normalization/attention policies, public training-job admission, LoRA and quantized-base training remain separate gates. The complete small-head diagnostic above now composes this policy but fails independent trajectory parity; released mixed-head certification remains pending.

Initial BF16 audit and remaining full-head prerequisites

Complete BF16 head training remains unsupported. The initial audit found that the trainer represented all floating tensors as FP32. The bounded projection implementation above adds typed BF16 saved operands while retaining FP32 master hydration, snapshots and updates. The planner recognizing a two-byte BF16 dtype does not establish a typed runtime ABI.

A controlled pre-fix reproduction supplied an FP32 layout and marked a trainable weight BF16. Codegen emitted a float* buffer, reported no skipped operations, and silently omitted the weight from parameter_gradient_order and optimizer updates. Storage admission now rejects unsupported tensor and reference dtypes before C emission. It also rejects inconsistent declarations, integer weights/gradients, and configurations claiming BF16 with FP32 tensors. The FP32 head author requires explicit FP32 arrays instead of interpreting packed uint16 weights as numeric FP32 values.

The following table records the initial audit; the projection milestone above resolves its bounded typed-cache case, not every full-head precision boundary.

BoundaryExisting evidence at auditBF16 composition gap identified
Generated training ABIFP32 buffers and INT32 metadata; FP32 updates and snapshots.Typed BF16 buffers, byte extents, initialization, saved tensors and checkpoint/export representation. See version/v7/scripts/codegen_train_runtime_v7.py.
Gradient synthesisExisting backward lowering constructs FP32 gradient and temporary tensors.Explicit gradient precision and conversion edges; no inferred casts from component names.
Span mean, exact-erf GELU, grouped poolingChecked BF16-storage VJPs in src/kernels/head_backward_bf16.c, three maps and independent leaf tests.The maps lack generated-training contracts. These variants consume and publish BF16 gradients; they cannot silently stand in for FP32-gradient variants.
Normalization and projectionBF16 LayerNorm and mixed MLP backward implementations exist in layernorm_kernels_bf16.c and mlp_kernels_bf16.c.Map/ABI/scratch and cache contracts for the actual head shapes. The fused MLP uses a tanh GELU approximation; Clef's exact-erf GELU cannot be replaced by it.
Noncausal attention and decision branchesFP32 generated-head derivatives; BF16 inference attention providers.Compatible BF16 training forward/backward, indexed/scalar branches and branch-gradient accumulation with explicit rounding points.
Optimizeradamw_update_bf16 exists with BF16 weights/gradients and FP32 moments.It is not the proposed FP32-master-weight recipe. Declare master/compute weight ownership and refresh boundaries before connecting updates.

Original implementation sequence (first step now demonstrated above): first prove typed storage and explicit conversion edges on one generic projection. The initial mixed-precision target uses BF16 projection operands, FP32 normalization/softmax/loss/scoring, FP32 master parameters, FP32 parameter-gradient accumulation and FP32 AdamW moments. Each provider retains its declared accumulation order; BF16 storage alone does not imply a particular compute instruction. Specify activation-gradient rounding separately and compare the exact declared operations against PyTorch; do not assume equivalence to arbitrary autocast settings. Then connect normalization, noncausal attention, pooling and scoring through the same graph. Certify the small complete head before repeating released-size tests. Existing BF16-gradient providers remain separately named variants rather than being silently repurposed.

The initial admission-only change fixed unsafe dtype handling and defined the missing boundaries; it does not claim a BF16 speedup, memory reduction, generated BF16 update or complete BF16 head PASS.

python3 -m unittest tests.test_v8_training_storage_contract -v
python3 scripts/nightly_runner.py --tests training_selective_parameters \
  training_complete_clef_head decision_head_backward_bf16 --ci --require-executed

The full-head oracle lane requires the pinned source described below. Standalone BF16 derivatives retain their numerical gates. Unsupported BF16 training must remain a clear error until typed execution and composed training evidence replace this restriction.

Released head weights: bounded FP32 training parity

The same canonical head topology now runs training at the released dimensions: 4,096-channel features, width 1,024, 16 attention heads, feedforward width 4,096, two evidence layers and four field layers. The head safetensors and independent reference source are checked against the pinned hashes before use. No kernel, derivative, compiler special case or numerical tolerance was changed.

Four native AdamW updates compare every effective parameter gradient, updated weight and optimizer moment against pinned PyTorch. On the fixed synthetic-feature request, loss falls from 0.59229439 to 0.13316447. Maximum absolute differences are 1.68e-7 for gradients, 9.54e-7 for weights and 2.36e-8 for moments. This measures arithmetic and state correctness, not downstream decision quality or generalization.

The fixture deliberately keeps backbone features and a small output-embedding table synthetic and frozen. It is released-head training with supplied features, not training with a generated backbone or reproducing the publisher's training data. FP32 AdamW uses the existing declared recipe: learning rate 0.001, epsilon 1e-4, weight decay 0.01, beta values 0.9 and 0.999.

A fresh subprocess loads the generated library, restores the checked checkpoint and executes the remaining native updates. Its loss sequence, final weight hash, optimizer-state hash and step count must exactly match uninterrupted execution. The existing same-process recovery, frozen-weight checks, rejection tests and separately generated inference export remain in place.

make build/libckernel_engine.so
CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py \
CKE_CLEF_FLASH_HEAD_PATH=/path/to/pinned/joint_head.safetensors \
  python3 scripts/nightly_runner.py --tests training_complete_clef_head \
    training_released_clef_head --ci --require-executed

The released-weight test has its own asset-backed nightly lane and a 600-second limit. Missing assets produce SKIP; configured files with incorrect hashes fail. The small 40-update trajectory remains a separate fast regression. This asset-backed certificate uses substantial memory and is not a latency benchmark.

Still unproven: generated-backbone feature training, a public dataset/job recipe, complete-head BF16, LoRA/QLoRA and real decision-task accuracy. The next precision milestone must certify storage, accumulation, gradients, optimizer updates and export separately.

Complete head topology: FP32 training on frozen supplied features

The bounded experiment now derives the training graph from the canonical complete-head circuit: two evidence-routing layers, four field layers, and every scoring branch. The existing shared v7/v8 backward synthesizer, memory planner, generated C and native AdamW execute 144 forward operations and 494 backward operations. Python authors the graph and runs the independent oracle; it does not update CKE parameters. No Clef-name branch is added to generic lowering.

On the small geometry (six tokens, eight channels, two questions with two/one options), all 162 effective trainable parameter arrays are compared with the hash-pinned reference class using PyTorch autograd. Forty updates reduce loss from 0.28329045 to 0.06633431. Maximum absolute differences over the trajectory were 1.38e-6 for gradients, 1.67e-6 for weights, and 3.15e-7 for optimizer moments (PyTorch 2.12.1 CPU, seed 611). The test checks gradients, weights, optimizer moments, byte-identical frozen feature/embedding weights, rejection without updates, exact checkpoint resume, and trained-weight reload through a separately generated inference entry. Overlapping spans and shared question indices exercise gradient accumulation.

The declared recipe uses FP32 AdamW, learning rate 0.001, weight decay 0.01, beta values 0.9/0.999, and epsilon 1e-4 in both implementations. This explicit epsilon limits amplification of roundoff-sized gradients on shift-invariant biases; it does not change the forward or gradient tolerance. The existing component recipes retain their own epsilon and remain separate regressions.

Reusable additions are checked indexed gather/scatter, explicit per-operation row counts for linear and add derivatives, generated integer metadata initialization checks, and opt-in contiguous flattening. The checked projection source is compiled with the generated library, using its existing scalar/AVX provider contract rather than changing global engine compiler flags.

make build/libckernel_engine.so
CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py \
  python3 -m unittest tests.test_v8_clef_full_training \
    tests.test_v8_indexed_rows_training -v

Standard nightly registers training_complete_clef_head. Missing reference source or PyTorch is explicitly SKIP; structural and native gather/scatter checks do not stand in for oracle execution.

Remaining gates: real generated-backbone features, a public dataset/job recipe, and complete-head BF16/LoRA/QLoRA training remain NOT_TESTED here. This fixed-feature low-level composition is not an unrestricted cke.v8.compile training API or an application-ready trainer.

Real-feature capture preparation — backbone run still NOT_TESTED

The next gate is a released-backbone feature capture followed by the packaged generated head. The local P3 cache currently contains the released head and reference source, but the post-trained Clef 9B backbone was not found. The new tooling does not download a model or substitute ordinary Qwen weights. Actual real-backbone capture and its generated-head numerical comparison remain NOT_TESTED until the pinned assets are supplied.

capture_clef_features.py is development-only PyTorch oracle tooling. It loads a local release after checking an administrator-pinned file-hash lock, committed head/config/source identities, and every indexed backbone shard. Tokenizer and other discoverable loader files must also appear in the lock. The lock must come from independently verified release assets; naming a revision in an untrusted manifest does not prove weight provenance. No production worker imports this script.

The pinned schema encoder silently truncates state at its length limit. This wrapper first encodes without truncation, rejects requests exceeding the declared capacity, then verifies the bounded result is identical. Images/video are explicitly unsupported. It retains original token IDs, question/option spans, stable candidate IDs and request hash. The backbone runs in BF16 with eager attention; actual hidden states and output-embedding rows are cast to FP32 for the same-input head comparison. This is not a claim of BF16 head parity or full-model FP32 parity.

Only used embedding rows are retained, with their original token IDs. The head-local IDs are remapped without changing their span positions or values. A nonzero complete-head PyTorch test proves that this compaction preserves every compared intermediate and final logit exactly. Capture loading checks byte hashes, token reconstruction, question ownership, expected reference shapes and finite reference values before native replay. Private requests/features remain local artifacts, not routine logs.

replay_clef_features.py builds a package using the existing authoring and compiler path, then compares four field-layer outputs, final normalized fields and all raw option logits. Intermediate absolute tolerance stays 1e-5; raw logits stay 2e-6, relative zero. It also compares each returned question's scores, retaining IDs and unequal option counts. Question zero → other questions → question zero tests selection and reuse within one captured context; it is not a substitute for distinct real-input A→B→A coverage. A package still has one compiled geometry. The existing 1,000 ms worker deadline is preserved; a deadline rejection is reported as execution failure with numerical NOT_TESTED, not silently bypassed.

# release-lock.json: {"model":"Cloudflare/clef-flash", "revision":"PINNED_REVISION",
#                     "files":{"config.json":"SHA256", ...every required local file...}}
# Obtain the lock and its expected SHA independently from verified release assets.
python3 -m version.v8.decision.capture_clef_features \
  --release /trusted/clef-release --release-lock /trusted/release-lock.json \
  --release-lock-sha256 LOCK_SHA256 --request request.json \
  --output /private/clef-capture --max-tokens 512 --threads 4
python3 -m version.v8.decision.replay_clef_features \
  --capture /private/clef-capture --capture-sha256 CAPTURE_JSON_SHA256 \
  --head /trusted/clef-release/joint_head.safetensors --output /private/clef-replay

# Cheap contracts; reference-source test skips explicitly when absent.
python3 -m unittest tests.test_v8_clef_feature_capture -v
# Real backbone gate is optional and must report SKIP without these assets:
CKE_CLEF_RELEASE_PATH=/trusted/clef-release \
CKE_CLEF_RELEASE_LOCK=/trusted/release-lock.json \
CKE_CLEF_RELEASE_LOCK_SHA256=LOCK_SHA256 \
python3 -m unittest tests.test_v8_clef_feature_capture.RealBackboneCaptureTest -v

The standard decision_feature_capture nightly lane registers both the cheap contracts and the explicitly asset-dependent real-backbone gate. Five local contract tests passed, including pinned-source truncation detection; the real-backbone test was skipped. This prepares evidence collection; it does not advance the CKE backbone, head training, LoRA or end-to-end decision certification.

Registered generated head: experimental loopback evaluation

The cke.decision.evaluate.v1 proposal now connects one registered generated head package to the independent simulated Antsand consumer over real HTTP. This is head_only / native_generated, not text-to-decisions. The existing synthetic fixture lane remains explicitly contract_simulation / fixture_replay. Neither executes business actions. The read-only cke.control.v1 protocol is unchanged; no real Antsand service or running deployment is modified.

BoundaryStatus
Supplied features → generated complete head → raw option logitsImplemented through a persistent loaded package and one generated native entry.
Released head weights over HTTP versus pinned referencePASS on P3; synthetic backbone features and embedding rows, two distinct inputs and A→B→A.
Text preparation / actual generated backboneNOT_IMPLEMENTED in this evaluation path.
Compute cancellation, remote authorization, real Antsand integrationNOT_IMPLEMENTED. Consumer cancellation suppresses publication; it cannot preempt C.
Head trainingExisting derivative/optimizer work is unchanged; this is forward inference integration.

The offline builder version/v8/decision/package_clef_head.py uses the ordinary circuit → map resolution → lowering → generated-C compiler. It packs effective FP32 weights and registered inputs; it performs no model arithmetic. Shared checked-graph build code now lives in version/v8/scripts/compile_checked_bundle_v8.py, with compatibility imports for existing tests. The worker uses only Python's standard library for request handling and byte copies; all head arithmetic remains in the generated entry. A subprocess test uses python -S to prove NumPy, safetensors and PyTorch are not runtime dependencies.

An administrator supplies a trusted package manifest and its expected SHA-256. This is executable native code, not an untrusted upload interface. The manifest binds the model revision, CKE revision, effective weights, circuit, layout, call IR, generated source, compiler/build identity, native library and registered feature files. The loader checks hashes, shapes, offsets, disjoint input/output/weight regions, finite features, token IDs, spans, final-valid-token selection, option counts and candidate ownership before loading C. On Linux, it loads the exact verified library bytes from a sealed memfd. Weights load once; each request clears non-weight arena regions and copies all of its registered inputs. The host does not schedule neural operations. Packages have one compiled geometry and at most 64 registered artifacts; runtime-variable feature shapes and arbitrary uploads are not supported.

Each registration contains hidden, input_ids, question_spans, option_spans, global_span, option_counts and question_type_ids, plus a selected question index and its stable candidate IDs. The entire multi-question context executes; one question's logits are returned. Reordering request candidate IDs only reorders those scores. Changing the head context requires a new registered artifact and hash. artifact_id in response evidence is the package SHA-256; ir_id is the call-IR SHA-256; library_sha256 identifies the loaded bytes. The input hash binds both feature bytes and the registration metadata. No host paths are returned.

The loopback worker allows one active evaluation, no evaluation queue and four connection threads. Concurrent evaluations return BUSY; excess connections close. A two-second absolute connection timer prevents slow clients from occupying threads indefinitely. Requests and responses remain bounded to 16 KiB, 16 candidate IDs and a maximum 1,000 ms consumer attempt. The worker checks expiration before and after native execution and before responding. The independent consumer rechecks the same absolute monotonic deadline inside its publication lock, after validation and lock wait. Cancellation/replacement invalidates pending results immediately. A disconnected or expired native call retains its slot and arena until C actually returns. Closing the worker joins handlers before unloading its library. Remote exposure, authentication and native interruption require separate contracts.

Reproduction

Prepare a local registrations JSON list with id, npz, question_index and candidate_ids. Each NPZ contains exactly the seven inputs above, hidden FP32 and other arrays int32, matching the declared geometry. Supply pinned released head weights and the intended output-embedding rows as an NPY. Record whether features and embeddings are synthetic or actual backbone captures; a pinned head alone does not establish either.

python3 -m version.v8.decision.package_clef_head \
  --head joint_head.safetensors --head-sha256 HEAD_SHA256 \
  --embedding embedding.npy --embedding-sha256 EMBEDDING_SHA256 \
  --registrations registrations.json --revision MODEL_REVISION \
  --output /trusted/head-package
python3 -S -m server.decision.native_server \
  --package /trusted/head-package/package.json --package-sha256 PACKAGE_SHA256
# Copy one printed bounded request object to request.json; use the printed loopback URL.
python3 -m server.decision.consumer --url http://127.0.0.1:PORT/evaluate --request request.json

OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
  server.tests.test_decision_native server.tests.test_decision_evaluate -v
python3 scripts/nightly_runner.py --tests decision_native_head decision_evaluate_contract --ci
# Optional independent released-head oracle lane:
CKE_CLEF_FLASH_HEAD_PATH=/path/joint_head.safetensors \
CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/joint_schema_model.py \
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
  server.tests.test_decision_native.PinnedNativeDecisionTest -v

The server prints fresh boot IDs and example requests; a restart invalidates old IDs. Do not treat this local descriptor as an approved remote discovery/authentication protocol. No persistent service is installed. Registered features may be private; retain packages accordingly.

Evidence and remaining gates

The asset-free nonzero synthetic head test compares returned logits to an independent PyTorch composition at absolute tolerance 2e-6, relative zero. It covers A→B→A versus a fresh runtime, candidate permutation, frozen weight bytes, sealed library bytes, invalid package identity/layout, native failure/recovery, BUSY, deadline expiration and cancellation during actual native execution. The existing fixture responder continues testing malformed protocol, restart and publication-lock behavior. Missing PyTorch or pinned assets are explicit SKIPs. Add --require-executed to nightly only with the required assets configured; it correctly fails on a missing-asset subtest even when the remaining tests pass.

On P3, released head/source identities from the committed Clef fixture were verified. Two distinct 6×4096 synthetic feature arrays, synthetic 16×4096 output embeddings, two questions and three options ran A→B→A through the packaged worker. Returned question-zero logits had maximum absolute error 2.39e-7 against the pinned direct reference (unchanged acceptance 2e-6); repeated A was exact. Native request execution was 234–265 ms in this run, excluding load/build. This is not a performance benchmark or end-to-end backbone result. Runtime responses always retain numerical NOT_TESTED: passing an independently run oracle test does not certify every subsequent request.

Next: agree on the execution contract with Antsand, register captured real backbone features and embeddings, then connect generated backbone execution. Keep head-only, full-model, transport, training and authorization verdicts separate. No decision score grants permission to act.

Indexed scoring gradients: separate normalization contracts

The final scoring branches now have checked FP32 derivatives for indexed pair features [a, b, a·b, |a−b|], normalized-vector dot products and cosine similarity. Owner indices route each option to its question. Repeated owners accumulate contributions; unreferenced owners receive zero. Both public gradient buffers are published only after all checks and reductions succeed. Caller-owned scratch holds groups·channels + 2·rows·channels floats; there is no kernel allocation.

Forward equivalence does not establish backward equivalence. The pinned head uses normalize-and-dot for its prior score and cosine_similarity for its joint score. PyTorch 2.12.1 normalization has a zero clamp derivative below epsilon, whereas its cosine implementation clamps a cloned norm under NoGrad and retains the norm derivative below epsilon. The new providers encode those policies separately. At zero norm, the norm VJP is zero; the direct numerator derivative still flows. At the normalization clamp boundary the derivative is one. Absolute difference uses zero subgradient at equality, including signed zero.

Consequently the cosine provider's below-clamp VJP deliberately matches the pinned framework's backward convention, not the finite-difference derivative of its clamped forward function. Central differences are tested away from nonsmooth/clamped boundaries; separate autograd tests cover zero, below epsilon, equality and adjacent FP32 values on both operands. Future head backward authoring must select the explicit derivative contract. Neither forward provider identity nor epsilon alone selects it.

Independent PyTorch 2.12.1 CPU comparisonLargest observed absolute gradient error
Indexed pair features2.39e-7
Normalized-vector dot2.24e-8
Cosine similarity2.24e-8

Measured 2026-11-08 on synthetic nonzero operands, widths 1, 7 and 129, with repeated and unreferenced owners. Primitive PyTorch ceilings are absolute 2e-6 / relative 2e-5; generated joined gradients use 3e-6 / 3e-5. These are empirical fixture bounds. Ordered scalar tests also cover width 64, O0/O3 equality and the reduction policy: each row contribution rounds FP32, then repeated-owner contributions sum in ascending row order in FP64 before the final FP32 store. An adversarial [1e8, 1, −1e8] scatter retains one; this does not claim bitwise equivalence to FP32 index-add.

One ordinary generated backward entry executes pair-feature/GELU and both cosine branches, then joins their left/right gradients using the existing checked addition provider. It compares the combined VJP against PyTorch, exercises A→B→A arena reuse, rejects insufficient arena capacity and a late nonfinite branch, and recovers. A deliberately miswired branch join produces a detectable gradient error. Primitive tests cover null/misaligned pointers, every short capacity, partial writable aliases, invalid owners, zero/overflow dimensions, invalid epsilon and late finite scatter overflow.

OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
  tests.test_v8_decision_scoring_backward -v
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
  --tests decision_scoring_backward --ci --require-executed

This is generated scoring-component backward, not full-head training. The circuit explicitly authors the derivative branches; automatic complete-head backward, scalar/gain and remaining indexed derivatives, grouped losses, all-parameter AdamW trajectory, BF16 optimizer certification and adapters remain separate gates. Existing full-head forward and attention-training regressions remain independent evidence. No Clef-specific compiler branch was added.

Noncausal attention backward and native training

A bounded generic composition now trains through feature lookup → trainable projection → Q/K/V projections → full noncausal attention → residual → scores → cross-entropy. Automatic backward lowering declares every branch accumulation and native AdamW updates the selected parameters. The new provider handles unequal query/memory lengths, packed token-major heads and caller-owned scratch. It does not implement masking, dropout or BF16 attention backward.

Forward arithmetic remains FP64 ascending dot products with FP32 score/exponential stores. Backward recomputes those scores; dS stores FP32, dQ reduces FP64, and shared dK/dV accumulate in ascending query order with an FP32 store after each query. All three gradient outputs are staged until validation and arithmetic succeed. Scratch is Q·H·D + 2·S·H·D + 2·S floats.

The shared trainer now propagates map-declared zero-success status. A checked kernel failure stops the step before AdamW, leaves weights/moments untouched, clears the incomplete accumulation window and leaves the caller's loss output untouched. This is not general rollback for arbitrary void providers. The fixed-batch contract rejects partial batches rather than allowing padding into unmasked softmax; native kernel tests cover unequal Q/S, while the composed training fixture uses four tokens. Variable-length masked training remains a separate contract.

FP32 native AdamW, 40 updatesInitial → final lossGradient / weight / moment max absolute error
All Q/K/V trainable1.32838511 → 0.000331591.75e-7 / 9.54e-7 / 5.22e-8
Frozen K projection1.32838511 → 0.000415671.20e-7 / 9.54e-7 / 3.73e-8

Measured 2026-11-07 with PyTorch 2.12.1+cpu and seed 837. The oracle uses explicit independent softmax attention and autograd, not Unsloth. Ordered scalar FP64/FP32 comparisons at O0/O3 and native forward finite differences provide additional checks. Gradient/weight ceilings remain 3e-5 absolute, 3e-4 relative; optimizer moments 3e-6 absolute, 3e-4 relative. These are bounded empirical checks, not a universal error bound or bitwise PyTorch claim. Reports retain engine, generated-C and runtime-contract hashes.

Checkpoint reset/resume and trained-weight inference reload pass. Frozen K weights remain byte-identical while gradients reach the upstream trainable projection. Swapping the saved K input with V fails the gradient comparison. An injected late backward failure after one successful microstep clears partial gradients; two subsequent microsteps exactly match a clean two-microstep baseline. Null/invalid IDs, labels, batch lengths, undersized capacities, nonfinite operands, aliases, misalignment and finite accumulation overflow have rejection coverage.

make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
  tests.test_v8_attention_cross_backward tests.test_v8_attention_training -v
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
  --tests training_attention_composition --ci --require-executed

Complete Clef-head training remains NOT_TESTED. This certifies a synthetic attention/projection/residual training composition. Next: connect existing head VJPs and the remaining indexed/scoring derivatives and grouped loss, then certify every released-head gradient and update. BF16 optimizer trajectories, LoRA and quantized-base training are separate milestones.

Cached-statistics LayerNorm training composition

The next trainable composition is fixed features → trainable projection → affine LayerNorm → trainable scores → cross-entropy. Existing native LayerNorm forward/backward arithmetic is reused; no new normalization kernel or model-specific compiler branch is introduced. The forward provider publishes its actual FP32 mean and reciprocal standard deviation, which automatic backward lowering passes to the existing VJP. Epsilon belongs to the operation, independently of the enclosing model.

Shared training code generation now accepts an opt-in, map-owned ABI for bounded contiguous FP32 buffers and literal scalar parameters. It rejects missing ports/providers, reordered ABIs, unsupported phases/dtypes, invalid scalars, insufficient capacities, incorrect shapes and writable aliases instead of guessing argument bindings. The selected maps participate in runtime/checkpoint identity. The legacy LayerNorm providers remain void: finite operands are their caller precondition. Checked providers now use the separate zero-success failure contract described above.

FP32 native AdamW, 40 updatesInitial → final lossGradient / weight / moment maximum absolute error
Trainable affine, epsilon 1e-51.29325175 → 0.004435791.98e-7 / 9.54e-7 / 5.59e-8
Frozen affine, epsilon 0.031.31046605 → 0.052932538.95e-8 / 6.26e-7 / 3.36e-8

Measured on 2026-11-07 against PyTorch 2.12.1 CPU, seed 837. Every trainable gradient, weight and AdamW moment is compared at every update. Frozen weights remain byte-identical; an upstream finite difference checks gradient flow. Resetting and restoring a step-20 v8 checkpoint reproduces remaining updates exactly, and final-weight reload reproduces forward-only logits. A deliberately swapped mean/rstd capture fails the gradient oracle. Direct native VJPs also cover widths 1, 7, 512, 513 and 4096. These are synthetic operands, not released Clef-head training or backbone features.

Numerical ceilings retain absolute 3e-5 and relative 3e-4 for gradients, weights and logits; moments use absolute 3e-6 and relative 3e-4. They are empirical fixture bounds. The existing forward uses compile-target-dependent FP32 SIMD variance partials and FP64 merging; backward reduces in FP32. Neither cross-compiler bitwise agreement nor BF16 optimizer certification is claimed.

make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
  tests.test_v8_layernorm_training tests.test_v8_selective_training -v
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
  --tests training_layernorm_composition training_selective_parameters --ci --require-executed

This change also aligns the shared v7 attention-score derivative scratch shape with the saved attention tensor, matching the v8 rule. It resolves the inherited attention-rematerialization shape assertion reported in the earlier selective-training sweep.

This does not yet make the complete Clef head trainable through cke.v8.compile. It extends the existing backend, maps and v8 derivative rules. Remaining work includes head authoring into the training IR, integration of the certified noncausal attention VJP, indexed/scoring derivatives, grouped losses and complete head parameter/optimizer/export comparisons. BF16 head trajectories and LoRA/QLoRA follow that baseline.

Automatic pooling derivatives and typed training inputs

Span means and grouped softmax-weighted pooling now use declared FP32 training contracts and v8 derivative rules. The shared lowerer generates their VJPs and explicitly accumulates gradients from separate branches. Integer spans and group counts are nondifferentiable; overlapping spans sum their feature-gradient contributions. Existing kernel arithmetic and inference ABIs are unchanged.

Fixed-batch training graphs declare config.training_integer_inputs, an ordered list of INT32 input tensor IDs. Generated summaries expose integer_inputs (tensor, offset, count, dtype) and integer_input_count. These bindings participate in runtime/checkpoint identity. The host packs metadata in that order and calls:

ck_train_step_with_i32_ex(token_ids, targets, valid_tokens,
                          integer_inputs, integer_count, loss_out, learning_rate)

The generated runtime copies metadata for the current batch and owns the forward/backward/optimizer schedule. The caller retains ownership of its source buffers. The old step entry cannot silently reuse metadata from a previous batch: it rejects graphs requiring these inputs. Forward-only inspection uses ck_train_set_batch_with_i32_ex followed by the existing generated forward entry.

Counts, alignment and exact input extents are checked before binding. Providers validate span ranges and positive group counts with exact option coverage. Compile-time checks reject missing metadata, trainable integer inputs, incompatible shapes/dtypes, wrong contiguous strides, and partial-token span training. This initial contract requires fixed capacities with all token rows valid; it does not certify arbitrary dynamic shapes. The existing generated runtime remains non-reentrant and must be serialized per loaded instance.

The regression composes fixed features → trainable projection → overlapping spans → exact-erf GELU → grouped weighted pooling → frozen projection → scores. The projection also feeds the grouped queries, so backward must add both branches. Six spans form unequal groups [2,1,2,1]. Forty native AdamW updates reduce loss from 1.40194964 to 0.16358216. Against PyTorch 2.12.1 CPU, maximum absolute differences are 8.01e-8 for gradients, 4.77e-7 for weights and 3.36e-8 for moments. Existing fixture bounds are unchanged.

Tests compare every trainable gradient, update and moment, upstream finite differences, frozen weights, same-process checkpoint restoration and inference reload. A wrong saved-query control fails the oracle. A→B→A inputs, caller-buffer mutation after binding, malformed metadata, nonfinite input and injected backward failure cover reuse and recovery. Admission rejection leaves the current accumulation intact; provider failure discards its partial accumulation window and recovery matches fresh execution. PyTorch absence is an explicit oracle skip.

make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_pooling_training -v
python3 scripts/nightly_runner.py --tests training_pooling_composition --ci --require-executed

This does not register complete Clef training. Indexed/scalar scoring derivatives, grouped decision loss, full-head training authoring and released-head gradient/update/export evidence remain. BF16 training, LoRA/QLoRA and application submission remain separate acceptance gates. The existing job service continues to reject unsupported recipes.

Automatic training through learned gains and combined scoring branches

The FP32 scoring composition now joins normalized-dot prior scores, cosine joint scores and a residual MLP, with two learned exponential scales and a sigmoid gate, before grouped decision loss. Its derivatives are selected by v8 rules and map contracts; the existing shared lowerer emits branch accumulation, parameter gradients and native AdamW. This milestone adds no generic compiler logic or model-specific backward scheduler.

Three reusable checked VJPs cover scalar exp-clamp, scalar sigmoid and vector scaling. Exp-clamp has derivative one at the clamp boundary and zero above it. Sigmoid differentiates the rounded stable forward value. Vector scaling computes input gradients and an ascending FP64 gain reduction, staging both outputs before publication. Caller-owned scratch, capacities, finite values, alias rejection and failure atomicity are explicit. Existing cosine/normalized-dot VJPs and checked addition gain training bindings. Their distinct below-epsilon norm derivatives remain explicitly selected; the forward result alone cannot identify which derivative is appropriate.

The nonzero model-neutral fixture uses two questions with one and three options, repeated owner indices and overlapping spans. All three learned scalars have significant initial gradients and independent finite-difference checks. Every trainable parameter gradient, weight and moment is compared with PyTorch through 40 native AdamW updates. The frozen feature table and redundant common score bias remain byte-identical. Checkpoint restore reproduces the remaining trajectory exactly, and trained-weight reload reproduces scores. An injected sigmoid-VJP failure after partial accumulation preserves weights and moments, discards incomplete gradients and recovers to the exact fresh-run update.

Measured on P3 with PyTorch 2.12.1 CPU, seed 837: loss 0.77345836 → 0.00013260. Maximum absolute errors were 2.38e-7 for gradients, 7.16e-7 for weights and 1.09e-7 for moments. Existing regression ceilings remain unchanged. Primitive tests cover independent ordered FP64 arithmetic, O0/O3, PyTorch, finite differences, saturation/clamp boundaries, finite overflow, one-element-short capacities, guards, partial overlap and rejection/reuse. The explicit FP64 gain test sums [1e8, 1, -1e8] to one.

The unchanged complete-head forward test also passes with hash-verified released head weights and pinned source at six and eight synthetic feature tokens: maximum raw-logit errors are 1.19e-7 and 1.79e-7. This protects forward inference; it does not certify released-head backward execution.

make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_scoring_training -v
python3 scripts/nightly_runner.py --tests training_scoring_composition --ci --require-executed

Complete released-head training remains NOT_TESTED. This is a reduced scoring graph, not the four field layers and evidence stack trained together. Next assemble those existing training components into complete head authoring, audit remaining broadcast/selection and trainable-embedding bindings, and certify every head gradient, update and export with released weights. No new application recipe is exposed. BF16 composed training, backbone execution, LoRA and QLoRA remain separate gates. The current trainer requires fixed capacities and positive scalar constants; the standalone exp-clamp kernel's finite-maximum domain is wider. Missing PyTorch or a native engine is reported as SKIP.

Declared grouped decision loss and indexed scoring training

A training graph can now declare its scalar loss and score tensor explicitly instead of requiring an operation named logits. The selected map declares the gradient input, gradient output, scalar value and reduction identity. The existing shared backward lowerer seeds that tensor and applies the normal derivative rules and branch accumulation. The existing generated native runtime performs forward, backward and AdamW; no model-name dispatch or separate decision training loop is added.

The new checked FP32 grouped cross-entropy provider accepts positive option counts summing exactly to the score count and one relative label per question. Loss is the mean over questions, including zero-loss singleton questions. It uses ascending FP64 max-shift exponential reductions and FP32 output stores. Loss and gradients publish only after validation. Scratch, typed INT32 labels/counts, fixed capacities and numerical identity are declared; BF16, ignore-index, weighting and double backward are not supported. Existing decoder graphs retain their implicit cross-entropy seed.

The small nonzero regression composes frozen features → trainable projection → overlapping spans and indexed pair features → biased projection → erf GELU → scores → grouped loss. It tests repeated owner indices, unequal option counts, all trainable gradients, an upstream finite difference, 40 native AdamW updates, frozen bytes, optimizer moments, checkpoint restoration and trained-logit reload. Invalid question counts or labels after a partial accumulation window must discard that window without updating weights or moments; subsequent updates must match a clean runtime.

The final scalar score bias is explicitly frozen: adding the same offset to every option cannot change cross-entropy. During bring-up, near-zero FP32 PyTorch reduction residuals caused AdamW to move that redundant bias differently (about 0.02246 on the first update). This fixture does not certify a trainable shared score bias. The hidden projection bias remains trainable. No tolerance was widened.

On P3 with PyTorch 2.12.1 CPU, the bounded fixture loss fell from 0.55899101 to approximately 3e-8. Maximum absolute errors across 40 updates were 1.94e-7 for gradients, 6.56e-7 for weights and 5.96e-8 for moments. The existing absolute ceilings remain 3e-5 for gradients/weights/logits and 3e-6 for moments, with relative 3e-4. Kernel tests also use an explicit FP64 scalar oracle, finite differences, O0/O3 comparison, padding and malformed-buffer rejection. Missing PyTorch or engine assets are skips.

make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_grouped_loss_training -v
python3 scripts/nightly_runner.py --tests training_grouped_loss_composition --ci --require-executed

This is component training, not complete Clef head training. Remaining work includes complete head training authoring (the combined scoring update above now covers scalar/gain and cosine derivatives), released-head gradient/update/export comparisons and the application recipe. BF16 composed training, backbone integration, LoRA and QLoRA remain separate gates. The job service still rejects unsupported Clef recipes. These low-level declarations do not yet make arbitrary inference circuits trainable through cke.v8.compile.

Exact-erf GELU in automatic backward lowering

The existing FP32 checked GELU forward and derivative now expose contiguous, fixed-batch training contracts. A v8 derivative rule declares the saved pre-activation, incoming gradient and scratch extent; the shared backward lowerer inserts the VJP and explicit gradient accumulation. There is no handwritten Clef backward scheduler and no new arithmetic kernel. The shared training library now links the existing GELU and head-VJP implementations.

Derivative rules may explicitly name their read ports, preserving existing provider ABIs. Legacy positional names remain supported. Duplicate, missing or invalid names are rejected. Strict non-partial lowering now rejects unresolved backward inputs instead of producing an incomplete training graph. Declared contiguous row strides are checked against tensor widths before C emission.

The model-neutral regression trains fixed features → projection → exact-erf GELU → frozen projection → option scores with the existing native cross-entropy and AdamW. It compares every trainable gradient, weight and moment with independent PyTorch, checks an upstream finite difference, frozen-weight identity, 40 updates, same-process checkpoint restoration and trained-logit reload. A late injected VJP failure must discard partial accumulation and preserve weights/moments; recovery must match a fresh run. This is a small FP32 composition, not released Clef training, BF16 training or a new job recipe.

Measured on P3 with PyTorch 2.12.1 CPU: loss 1.40563822 → 0.18035129 over 40 native updates; maximum absolute differences were 9.32e-8 for gradients, 7.16e-7 for weights and 4.48e-8 for moments. The unchanged fixture ceilings are absolute 3e-5 for gradients/weights/logits and 3e-6 for moments, with relative 3e-4. PyTorch absence explicitly skips the independent comparison.

make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_gelu_training -v
python3 scripts/nightly_runner.py --tests training_gelu_composition --ci --require-executed

Complete Clef training remains unsupported. The complete inference circuit still now has indexed/scalar and grouped-loss component evidence; complete graph authoring remains. Training authoring must then connect that graph to the existing workflow. Only after complete gradients, updates and export/reload pass should the local job registry advertise a Clef head recipe. Backbone-feature capture and frozen-backbone execution remain separate evidence levels.

Selective FP32 training prerequisite

The shared v7/v8 backward lowerer now honors boolean parameter requires_grad declarations without cutting activation gradients through a frozen operation. Conflicting declarations on the same shared weight are rejected before emission. Undeclared weights retain legacy trainable behavior. Operation differentiability remains separate from parameter trainability: a frozen projection between trainable operations must still participate in backward execution.

Frozen parameters have no canonical parameter gradient, optimizer moment allocation or AdamW update. Existing fused GEMM backward providers still require temporary dW/db output buffers, so these derivatives are computed into discard scratch. This is a correctness contract, not a claim that all unnecessary backward arithmetic or scratch has been eliminated.

A small generic linear decision fixture executes fixed feature lookup → trainable projection → frozen projection → trainable scores → cross-entropy through the existing training IR, memory planner, generated C and native AdamW. This fixture is not the complete Clef head and is not a new public feature-head authoring API. No Python code updates CKE weights.

On 2026-11-07, 40 FP32 native updates reduced loss from 1.42433834 to 0.04014646. Every trainable gradient, updated weight and optimizer moment was compared with independent PyTorch CPU execution. An upstream finite difference verifies gradient flow through the frozen middle projection; feature and bridge weights remain byte-identical. The existing v8 checkpoint format restores step 20 into a reset runtime and reproduces the remaining 20 steps exactly. Reloading final weights reproduces inference logits. Observed maximum absolute errors against PyTorch were 7.08e-8 for gradients, 7.31e-7 for updated weights and 2.24e-8 for moments. Regression ceilings are absolute 3e-5 for gradients/weights/logits and 3e-6 for moments, with relative 3e-4. These are bounded fixture contracts, not universal error guarantees. This test does not claim fresh-process resume or released-head training. PyTorch or native-library absence is an explicit skip.

make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 tests/test_v8_selective_training.py
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
  --tests training_selective_parameters --ci --require-executed

Eight focused tests and the registered nightly lane pass. Both existing decoder training compositions (2l_dense, 2l_gqa_tail) pass gradients, ten updates, checkpoint/resume and inference export. A broader 47-test sweep had 46 passes and one attention-rematerialization shape assertion failure; the same failure was reproduced with the unchanged main lowerer. Full repository nightly has not been run locally.

Next: connect the remaining complete-head derivatives and grouped loss to this existing training workflow, then certify head gradients, updates and export. BF16 VJP providers are independently tested; this FP32 optimizer trajectory does not certify BF16 head training. LoRA and quantized frozen-base training remain separate, later gates.

Complete bounded head forward pass

build_clef_flash_complete_head_circuit.py executes 140 operations in one generated entry: head ingress → two evidence-routing layers → option summaries → field assembly → four pre-norm self/cross-attention decoder layers → field/option normalization → prior, cosine and residual scoring → raw option logits. The host supplies hidden features, output-embedding rows, token IDs, checked spans, option counts and question types. It does not schedule the layers.

Question types index the released three-row type embedding. Consecutive option groups retain their question/option order; tests reorder entire questions with their option blocks and type identities. No softmax decision policy is imposed on the final raw scores. There is no generated-backbone or complete service certification in this milestone.

Six reusable checked operations close the scoring gaps: row broadcast addition, indexed pair features (left, right, product, absolute difference), indexed cosine reduction, clamped scalar exponential, stable scalar sigmoid and scalar vector multiplication. Their maps declare sizes, scratch, numerical semantics and status propagation. Generic compiler changes are operation catalogue entries only.

Pinned head, synthetic inputsWorst decoder-block absolute errorFinal logit maximum / RMSE
6 tokens, counts [2,1]3.81470e-61.19209e-7 / 9.73340e-8
8 tokens, counts [1,2]5.72205e-61.78814e-7 / 1.06067e-7

Measured 2026-11-07 with PyTorch 2.12.1 CPU, the hash-verified released head and its direct forward implementation. Both sides receive identical synthetic hidden features and synthetic output embeddings. These are two separately compiled bounded geometries, not a dynamic serving interface or captured backbone trajectories. Intermediate absolute regression ceilings are 1e-5; final logits use 2e-6. These empirical bounds do not establish training or other precision contracts.

python3 version/v8/decision/build_clef_flash_complete_head_circuit.py --check
python3 -m unittest tests.test_v8_decision_tensor_ops tests.test_v8_clef_flash_complete_head -v
export CKE_CLEF_FLASH_HEAD_PATH=/path/to/joint_head.safetensors
export CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/joint_schema_model.py
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
  --tests decision_tensor_ops decision_clef_flash_complete_head --ci --require-executed

The independent comparisons require PyTorch; pinned comparisons additionally require safetensors, the reference source dependencies and the two hash-checked assets. Missing dependencies/assets are explicit skips. Synthetic tests cover intermediate logits, two lengths, unequal groups, question permutation, malformed IDs/spans/counts/type IDs, nonfinite hidden input, undersized arena, a deliberately corrupted scoring weight, and A→B→A recovery. A model-neutral six-operation generated circuit exercises all new providers independently of Clef topology.

First generated backward composition

Three reusable FP32 vector-Jacobian products now cover token-span means, exact erf GELU, and grouped query-conditioned weighted pooling. They are backward providers, not inference aliases. Their maps own strides, capacities, scratch, numerical contracts and status propagation. The checked-call emitter now accepts an explicit backward phase and rejects providers used in the wrong phase. No model-name dispatch or second training workflow is introduced.

The model-neutral regression executes one generated forward entry for span pooling → erf GELU → grouped pooling, then one generated backward entry for the reverse chain. Only native-produced forward intermediates enter that backward entry. This is an explicitly authored VJP circuit, not automatic backward synthesis through cke.v8.compile. It does not contain trainable parameters, a loss, optimizer updates, or the remaining complete-head derivatives.

  • Span gradients sum all overlapping contributions in ascending span order with FP64 accumulation. Unreferenced valid tokens become zero; inactive token rows and padding remain unchanged.
  • Erf GELU uses its analytic derivative in FP64, distinct from the existing tanh-approximation derivative.
  • Grouped pooling differentiates both value and score paths, recomputing the forward provider's FP32 probabilities. Ordered FP64 reductions and explicit FP32 intermediate stores are declared in its map. The options and query gradients publish together only after all checks succeed.
  • Each VJP overwrites its valid output gradient. Overlapping contributions inside a VJP accumulate; fan-out across separate graph operations still requires an explicit gradient-accumulation operation.
  • All outputs and scratch must be disjoint from read buffers and one another. Scratch may change on rejection; that kernel's gradient outputs do not. A later operation failure can leave earlier graph intermediates written, so the entry status must be checked. Tests inject a final-operation failure and verify that the final token gradient remains untouched, then recover.

Tests cover independent ordered arithmetic oracles, optional PyTorch autograd, finite differences for both grouped-pooling inputs, unequal and singleton groups, padded strides, null/misaligned/overlapping regions, capacity and finite-overflow rejection, and A→B→A recovery. Generated execution and scalar oracles require no model assets or PyTorch. PyTorch absence is an explicit SKIP for the autograd comparison.

OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_head_backward -v
python3 scripts/nightly_runner.py --tests decision_head_backward --ci --require-executed
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 version/v8/scripts/run_training_composition_v8.py \
  --run-root version/v8/.cache/clef-backward-decoder-regression
FP32 gradient versus PyTorch 2.12.1 CPU (2026-11-07)Maximum absolute errorRMSE
Overlapping-span input2.98023e-86.23360e-9
Erf GELU input5.96046e-83.12435e-8
Grouped pooling options (both paths)4.47035e-81.70069e-8
Grouped pooling query1.02445e-84.22744e-9

These are measured small, nonzero synthetic cases, not universal error bounds or released-head gradient certification. Regression comparisons use absolute 4e-7 / relative 3e-6 against PyTorch, and exact checks for the explicitly ordered span/grouped arithmetic oracle. The focused 9-test suite passes; the four selected standard-nightly suites pass with no skips, including the hash-checked released-head forward comparison. Both existing decoder compositions (2l_dense, 2l_gqa_tail) also pass 10 native AdamW updates, trajectory/gradient/weight/moment comparisons, fresh-process resume and inference export. Their aggregate report was /tmp/clef-backward-decoder-regression.json. The full repository nightly is not a local result.

Complete Clef head training remains NOT_TESTED. Next integrate checked biased-linear, LayerNorm and noncausal attention derivatives, indexed/scoring derivatives and branch accumulation into the existing v8 backward workflow. Then certify all head gradients, grouped loss, AdamW updates, frozen-backbone identity, checkpoint/resume and generated inference reload. Existing decoder training is a shared-consumer regression, not evidence that this head can already train.

BF16-storage backward providers

The same three VJPs now also have explicit BF16-storage maps. Saved floating-point operands, incoming gradients and outgoing gradients use BF16; spans/counts remain INT32. Caller-owned FP32 workspace expands BF16 operands exactly, invokes the existing FP32 VJP arithmetic, then rounds each output gradient to BF16 using round-to-nearest-even. The FP64 reductions and FP32 intermediate stores inside the FP32 providers are preserved. There is no intermediate BF16 rounding inside a VJP.

This is a particular mixed-precision derivative contract, not a claim of parity with an arbitrary BF16/autocast forward graph. Grouped probabilities are recomputed using the existing FP32 forward semantics on expanded BF16 saved operands. Independent autograd runs FP32 computation on identical dequantized operands and rounds its gradients to BF16. The tested gradients match that oracle exactly after rounding; this is not a universal bit-identity guarantee.

ProviderRequired FP32 workspace (floats)
Span mean(span_count + token_capacity + valid_tokens) × channels
Erf GELU4 × rows × channels
Grouped weighted pooling3 × option_rows × channels + 6 × groups × channels + 2 × option_rows

Element counts and row strides use the pointed-to dtype, not bytes. BF16 buffers require two-byte alignment; INT32 spans/counts and FP32 scratch require four-byte alignment. Outputs and scratch must be disjoint from all reads and each other. Input padding is ignored, output padding/inactive rows stay untouched, and rejected calls publish no outputs. In particular, a finite FP32 gradient that rounds to BF16 infinity rejects the call before either grouped gradient is published.

The model-neutral generated backward entry composes all three BF16 providers through ordinary maps, lowering and memory planning. Tests include odd channel counts, unequal/singleton groups, ordered oracles at O0/O3, exact/one-short capacities, two-byte-aligned buffers, mixed-width partial overlaps, nonfinite input, FP32 and BF16-only overflow, RNE midpoint ties, subnormal values, final-operation failure propagation, and A→B→A recovery. Saved inputs in this test are synthetic; this does not execute a complete BF16 Clef forward/backward pair. PyTorch absence skips only the independent autograd test.

OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_head_backward_bf16 tests.test_v8_head_backward -v
python3 scripts/nightly_runner.py --tests decision_head_backward_bf16 decision_head_backward --ci --require-executed

Local validation on 2026-11-07: nine BF16 tests and 138 combined BF16/FP32, ABI, numerical-resolution, provider-selection, ledger and documentation tests pass. Three selected standard-nightly suites pass without skips, including the pinned complete FP32 Clef head forward comparison. DSL policy and interface/allocation audits pass. Full repository nightly and BF16 full-head training were not run.

Correctness baseline, not a speed claim. These portable conversion adapters need no BF16 ISA but add conversion work and workspace. Benchmark a complete training step before claiming latency or peak-memory improvement. Complete BF16 head training, gradient accumulation precision, FP32 master weights/optimizer state, loss scaling policy and checkpoint dtypes remain separate contracts.

Next training evidence

The next completion gate is a small nonzero trainable head composition through the existing v8 workflow: verify every parameter gradient, demonstrate several native AdamW updates and loss reduction, prove frozen-weight identity, then save/reload and reproduce the updated scores. Expand this to the complete head once normalization, noncausal attention, projection and scoring derivatives are wired. The current primitive tests do not establish a trained decision model.

After FP32 head training, certify BF16 training and declarative LoRA independently. Quantized LoRA requires a selected frozen-base format with a tested backward-input path; inference Q4/INT8 kernels alone do not establish QLoRA. Keep model topology/trainability in graph declarations and optimizer arithmetic in the existing generated native workflow.

Backward audit and next training gate

Shared-consumer check on 2026-11-07: run_training_composition_v8.py completed both 2l_dense and 2l_gqa_tail with PASS. These existing generated decoder training certifications do not certify Clef backward execution.

Forward passes; head training remains NOT_TESTED. The full forward head is not yet linked to automatic backward execution. Extend cke.v8.compile and the existing generated-C optimizer, checkpoint/resume and inference-export workflow. Do not add a second model executor.

Head computationExisting implementationMissing contract/certification
Linear projections, biasesgemm_backward_f32.json maps fc2_backward_kernel; gemm_backward_bf16_mixed exists in gemm_kernels_bf16.c.Bind checked head geometry and saved tensors; certify dX, dW and db, selective parameter gradients and accumulation.
LayerNormlayernorm_backward_kernel in layernorm_kernels.c; BF16 variant in layernorm_kernels_bf16.c.Cached-statistics training maps and native optimizer composition now pass; wire them into the complete head while preserving operation-local epsilon.
Erf GELUChecked erf derivative now has FP32 VJP and generated-chain evidence; earlier gelu_backward_exact differentiates tanh.Wire the erf provider into automatic head backward; BF16 remains separate.
Unmasked self/cross-attentionChecked FP32 noncausal dQ/dK/dV with ordered oracle, PyTorch autograd and native optimizer composition now pass.Integrate the provider into the complete head. Masked and BF16 attention training are separate; existing causal providers are not substitutes.
Token-span pooling and last-token selectionChecked span VJP accumulates overlapping spans with generated-chain evidence.Automatic span VJP and branch accumulation pass in the typed-input training composition; integrate final-row selection and the complete head.
Embedding/gather and residual branchesembedding_backward_f32.json, native BF16 embedding scatter, residual_add_backward_f32.json, gradient-accumulation providers.Wire type-embedding gradients, indexed-owner scatter and broadcast reductions. Frozen output embeddings need no updates, while required hidden-feature gradient paths remain explicit.
Grouped softmax-weighted poolingChecked options/query VJP covers both score and value paths, unequal and singleton groups.Automatic VJP and shared-input branch accumulation pass in generated training; connect complete-head authoring and loss.
Cosine, pair features, scalar transformsIndexed pair features and distinct normalized-dot/cosine FP32 VJPs now have generated branch-join evidence.Automatic FP32 combined scoring now covers pair/cosine branches and learned scalar/gain gradients with 40 native updates. Complete released-head authoring and training remain uncertified.
Loss and optimizersoftmax_cross_entropy_loss.json and adamw_update_f32.json; existing trajectory/resume/export machinery.Declare one label per question and loss reduction over unequal option counts, then verify every trainable parameter, frozen hashes, optimizer moments and restart equivalence.

version/v8/scripts/run_training_workflow_v8.py:_apply_semantic_model currently admits FP32, bias-free SwiGLU/RMSNorm decoder configurations at selected depths. Clef requires biased linear operations, LayerNorm, erf GELU, branches, span pooling and grouped outputs. Removing that validation alone would not implement their backward graph. Extend the responsible contracts while retaining the existing decoder certification suite.

Start with FP32 head-only training on a small nonzero graph and frozen backbone features. Compare loss, feature gradients, every head parameter gradient, one AdamW update, a short deterministic trajectory, checkpoint/resume and inference reload. Then use released head geometry. BF16 kernel availability is not composed BF16 head-training evidence; BF16, adapters and quantized-base training remain separate later gates. End-to-end backbone decisions also need actual generated features and output embeddings, rather than synthetic ones.

Earlier option-summary milestone

build_clef_flash_option_summary_circuit.py extends the canonical ingress and both evidence-routing layers through grouped option pooling and option_summary_norm. Its single generated entry executes 43 operations. Question option counts partition consecutive routed option rows. Each question's projected vector supplies its pooling query. Normalization declares epsilon 1e-5 locally, independently of the enclosing model configuration.

The grouped provider validates positive counts and exact coverage, finite valid inputs, physical strides, capacities, alignment and disjoint writable buffers. Caller-owned scratch stages the result, so rejection does not publish partial output. Read-only buffers may alias each other. Padding is neither consumed nor written. Arithmetic follows the registered ordered FP64 reductions with FP32 intermediate storage; no backward capability is advertised.

Reproduction

python3 version/v8/decision/build_clef_flash_option_summary_circuit.py --check
python3 -m unittest tests.test_v8_grouped_weighted_rows tests.test_v8_clef_flash_option_summary -v
python3 scripts/nightly_runner.py --tests decision_grouped_weighted_rows decision_clef_flash_option_summary --ci --require-executed

Use a Python environment with NumPy; independent oracle tests additionally need PyTorch. The released-head comparison also needs safetensors and the dependencies imported by the pinned reference source. Enable that comparison with:

export CKE_CLEF_FLASH_HEAD_PATH=/path/to/joint_head.safetensors
export CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/joint_schema_model.py

Both assets are hash-checked against tests/fixtures/decision/clef_flash/identity.json. Missing oracle packages or assets are explicit test skips. The reference test runs the pinned head's actual forward method and captures its summary normalization boundary; it does not inject reference evidence outputs into CKE. The host supplies synthetic hidden states, synthetic output-embedding rows, IDs and spans. Both implementations use the same inputs and released head weights. This is not a generated-backbone or complete-decision test.

On 2026-11-07, PyTorch 2.8.0 CPU comparisons passed for six tokens with option counts [2,1] and eight tokens with [1,2]. Maximum absolute errors were 1.90735e-6 / 1.43051e-6 for raw summaries and 1.31130e-6 / 1.19209e-6 for normalized summaries; normalized RMSE was 2.63926e-7 / 3.13428e-7. These are fixed compiled geometries, not proof of a dynamic serving interface. The declared pinned-prefix absolute regression ceiling is 1e-5.

Synthetic tests also cover reordered options within groups, A→B→A reuse, invalid counts before parity, undersized arenas, and rejection followed by successful execution. Model-neutral tests compare explicit FP64/FP32-stage arithmetic at O0/O3, including singleton groups, unequal groups and padded strides.

Related: kernel architecture, training capability ledger.

Image
100% | |
Scroll to zoom | Drag to pan | W/H to fit | 0 to reset | ESC to close