Clef-Flash decision-head bring-up
Registered local LoRA jobs
Register an approved package to enable decoder_lora_fp32_v1 through
cke.training.jobs.v2. The recipe accepts two-layer dense FP32 BUMP
decoders with rms_eps=1e-5, 2–64 fixed tokens, embedding width up to
256 and vocabulary up to 4096. An administrator supplies the package, base files,
and tokenizer identity. The trusted local adapter supplies the caller's identity.
Register a package
Save this as package.json alongside the relative directory containing
weights.bump and weights_manifest.json. Use actual artifact
hashes and label generated weights synthetic. Both splits must contain
records and cannot share identical input token sequences. The package is limited to
8 MiB and 4,096 records. Tokenize with the pinned tokenizer before submission.
{
"schema": "cke.lora.training-package.v1",
"base": {
"directory": "base",
"manifest_sha256": "<64 lowercase hex digits>",
"weights_sha256": "<64 lowercase hex digits>",
"provenance": "pretrained"
},
"tokenizer": {"id": "<pinned tokenizer ID>", "sha256": "<tokenizer hash>"},
"tokens": 4,
"records": [
{"id": "train-1", "split": "train", "tokens": [0,1,2,3], "targets": [1,2,3,4]},
{"id": "validation-1", "split": "validation", "tokens": [0,1,2,5], "targets": [1,2,5,4]}
]
}
Map an opaque package ID to its local file and SHA-256 in the registry:
{"demo": {"path": "/absolute/package/package.json", "sha256": "<package hash>"}}
Launch the local protocol from a clean CKE checkout with a built engine:
python -m server.training.jobs --lora-registry /absolute/registry.json \
--store /absolute/private-job-store --principal studio
Prepare, submit and retrieve
- Discover
executable_recipes, package identities, split sizes and projection targets. Pin the worker boot identity and capability hash. - Prepare with a 32-hex client revision, experiment ID and configuration. Review the returned plan and hash before submitting.
- Submit that hash with the same revision and an owner-scoped idempotency key.
- Observe progress and terminal status. After a worker restart, an interrupted
job reports
recovery_required. The configured wall deadline bounds a job. - Retrieve the report, adapter ZIP and checkpoint ZIP by artifact ID. Verify
the byte count and SHA-256. Keep the matching base and compiled runtime for
LoRASession.load_adaptersorload_checkpoint.
{
"recipe": "decoder_lora_fp32_v1", "package_id": "demo",
"package_sha256": "<package hash>", "dtype": "fp32",
"optimizer": "adamw", "method": "lora", "steps": 40,
"learning_rate": 0.0003,
"adapters": [
{"target":"layer.0.wq","rank":2,"alpha":4,"name":"query","seed":41},
{"target":"layer.1.wv","rank":3,"alpha":6,"name":"value","seed":42}
]
}
Generated C runs forward, loss, backward and AdamW. Successful jobs check
frozen weights and exact validation loss and logit hashes after both reloads.
Job reports label numerical oracle certification NOT_TESTED and
held-out quality MEASURED_NOT_CERTIFIED. This local protocol requires
a trusted caller. Remote authentication, cancellation and submitting a
checkpoint as a new job are outside the supported contract.
CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q \
tests/test_v8_lora_jobs.py server/tests/test_training_jobs.py \
server/tests/test_training_bridge.py server/tests/test_training_contract.py \
tests/test_v8_lora_session.py tests/test_v8_decoder_lora_training.py
python scripts/nightly_runner.py --tests training_job_lifecycle --require-executed
Bounded notebook LoRA training interface
The public cke.v8.prepare_lora interface accepts an existing FP32
BUMP base artifact and explicit LoRAConfig targets. It resolves a
checked plan, freezes all base parameters and prepares externally initialized
adapter values before compilation. plan.compile uses the existing
shared backward lowering, memory planner and native compiler. No Python gradient
calculation, optimizer update or second neural execution schedule is introduced.
plan = cke.v8.prepare_lora(base_directory, tokens=4, adapters=[
cke.v8.LoRAConfig("layer.0.wq", rank=2, alpha=4, name="query", seed=42),
cke.v8.LoRAConfig("layer.1.wv", rank=3, alpha=6, name="value", seed=43),
])
with plan.compile(new_run_directory, dataset_id="my-dataset-sha256",
max_steps=1000, learning_rate=0.003) as session:
metrics = session.train_step(token_ids, target_ids)
validation = session.evaluate(validation_ids, validation_targets)
session.save_checkpoint(checkpoint_directory)
session.export_adapters(adapter_directory)
cke.v8.lora_capabilities() reports the supported scope.
cke.v8.lora_targets(base_directory, tokens=4) discovers supported
projection IDs, layer/role, shapes, legal rank ranges and parameter costs without
initializing adapters, compiling or executing training. Initially:
two-layer dense qwen3-template FP32 artifacts with explicit RMS/QK epsilon 1e-5,
2–64 tokens fixed at compile time,
explicit projection targets, adapter-only native AdamW, and bounded workspace and
step count. It does not load arbitrary pretrained Qwen/GGUF models, implement
BF16 whole-decoder training or QLoRA, or accept arbitrary PyTorch modules. Token
IDs must match the base tokenizer; tokenization is not guessed.
LoRASession.open verifies artifact/build identities and creates
isolated native state. Adapter reload requires the same base, shapes, targets and
scales and a fresh optimizer. Checkpoint resume instead restores the complete
optimizer trajectory and declared dataset/step-budget identity. All checkpoint
payloads are validated before import. Calls on one session are serialized.
Tests execute the notebook cells, two concurrent session objects, input rejection and recovery, corrupt checkpoint rejection, incompatible adapter rejection, 40 native updates, and fresh-process reload without importing PyTorch or Transformers. Numerical certification remains in the complete-decoder oracle lane. Training loss is separate from held-out quality and performance.
Examples: version/v8/notebooks/03_generated_lora_session.ipynb and
version/v8/examples/python_lora_session_v8.py. Both require an explicit
existing base artifact and tokenized records; neither substitutes a synthetic
base or silently changes unsupported input. The notebook regression uses a
labelled synthetic base solely as test evidence.
make -j4 build/libckernel_engine.so CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q tests/test_v8_lora_session.py CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python scripts/nightly_runner.py --tests training_lora_fp32 --require-executed
Complete small decoder with FP32 adapters
attach_lora_adapters(..., freeze_base=True) explicitly freezes
all base parameters while preserving activation gradients through the decoder.
The default preserves existing selective-training behavior. Initialization remains
external; generated native code performs forward, backward and AdamW updates.
The regression uses the standard two-layer dense Qwen-style decoder template, including embeddings, normalization, RoPE, causal attention, residuals, SwiGLU and vocabulary loss. Rank-2 query and rank-3 value adapters in different layers total 320 trainable parameters. Four- and seven-token cases use separate compiled geometries; this does not certify dynamic lengths in one library.
Both cases compare named forward tensors, loss, every adapter gradient, weights and optimizer moments against independent PyTorch for 40 updates. Tests cover frozen-base bytes, zero-B initialization, finite differences, capacity/nonfinite rejection, checkpoint continuation and fresh-process forward-only reload. Missing dependencies remain explicit skips; numerical tolerances are unchanged.
Training loss fell from 4.764195 to 1.113996 and from 3.932966 to 1.229868. Maximum observed gradient error was 7.67e-6, weight error 2.55e-6 and moment error 4.49e-7. This is synthetic training correctness, not held-out application accuracy. Pretrained Qwen fine-tuning, base-import adapter manifests, complete BF16 decoder training, QLoRA and performance certification remain untested.
make -j4 build/libckernel_engine.so CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q tests/test_v8_decoder_lora_training.py tests/test_v8_lora_training.py tests/test_v8_lora_bf16_base.py tests/test_v8_selective_training.py
The existing training_lora_fp32 nightly lane includes this test.
PyTorch runs only as the independent development oracle.
Declared frozen native prefix in training
A checked feature-producing component can feed the existing generated head
trainer when its circuit explicitly declares stop_gradient=True.
Kernel maps declare a frozen forward boundary and borrowed context arguments;
shared lowering owns feature and scratch storage. Generated C executes the
component, head forward, backward and AdamW. Python need not transfer hidden
features or schedule component tokens.
The caller binds an exclusively owned, live native descriptor through
ck_train_set_native_context_ex and retains it until execution ends.
Contexts are not parameters or checkpoint contents and must be rebound after
initialization or reset. Changed inputs, weight imports, failed execution and
evaluation invalidate permission to apply stale gradients. Failed native
execution cannot advance the optimizer.
Small non-Clef tests compare five FP32 updates and parameter gradients against
PyTorch, including a declared FP32 LoRA adapter after the frozen component. They
test invalid IDs, nonfinite output, cancellation and recovery, and run a
relocated C-only training host after deleting its build directory. This certifies
the bounded interface. A separate released-model run executes the frozen 9B
backbone and one generated head update with no Python hidden-feature handoff.
Fresh pinned PyTorch head comparisons on the identical native features pass:
maximum gradient error 1.02e-7, weight error 2.38e-7 and
optimizer-state error 1.01e-8, preserving the existing tolerances.
A separate C-only host outside the checkout reproduces the complete updated
weight array and loss exactly, without a Python process or captured hidden inputs.
Loaded native library identities are retained; this is not arbitrary-machine portability.
See bound evidence.
This fixed-request experiment is not yet exposed by the application training
recipe. No backbone derivative, BF16 adapter or quantized-base training is advertised.
Connected backbone/head numerical acceptance remains a separate open gate.
python3 -m unittest tests.test_v8_frozen_component_training -v
BF16 compute in a frozen base, FP32 adapters
The existing projection-precision transformation composes with FP32 LoRA.
Attach the adapter first, then select the frozen base operation for
use_bf16_projection. Adapter matrices, base parameter storage,
gradients and AdamW moments remain FP32. The base projection saves RNE BF16
compute operands and accumulates in FP32; its input gradient still reaches
trainable operations upstream.
from version.v8.training.lora import attach_lora
from version.v8.training.bf16_projection import use_bf16_projection
adapted, spec = attach_lora(authored_training_ir,
target="bridge", name="task", rank=2, alpha=3)
base = next(op for op in adapted["ops"] if any(
ref["tensor"] == "weight.bridge" for ref in op.get("weights", {}).values()))
mixed = use_bf16_projection(adapted, base["op_id"])
# Prepare checked weights, then use the existing backward lowering/runtime.
The resolved BF16 projection providers are ck_projection_bf16_train_forward and ck_projection_bf16_train_backward in src/kernels/projection_bf16_train.c. They use portable C conversion and ordered FP32 arithmetic, with no BF16 hardware-ISA dispatch; these tests do not claim AVX-512 BF16 or AMX acceleration. Generated provider identities are asserted alongside native execution.
A fourth composition trains rank-two and rank-one adapters across two BF16-compute frozen projections. It requires the shared composition audit alongside independent gradients, unchanged FP32 base/scale bytes and exact fresh-process resume.
Three small generated compositions cover adapter-only training, an upstream
trainable projection, and rectangular dimensions. Each compares loss, gradients,
weights and optimizer moments for 40 AdamW updates, including checkpoint reload,
fresh-process continuation and inference reload. The independent PyTorch oracle
implements this declared saved-BF16-operand/FP32-gradient contract; it is not
ordinary PyTorch autocast. Frozen matrices and adapter scale remain unchanged.
Finite differences are not used to certify a straight-through derivative of
rounding. Retained evidence
records maximum gradient error 4.10e-7, weight error 1.67e-6
and moment error 7.81e-8 across these trajectories.
This reuses existing providers without a compiler or arithmetic change. It is not BF16 adapter storage, whole-model mixed precision, QLoRA or released Clef adapter certification. Base storage remains FP32 and saved BF16 buffers are additional memory; no speed or memory improvement is claimed without measurement.
python -m pytest -q tests/test_v8_lora_bf16_base.py
python scripts/nightly_runner.py --tests training_projection_bf16 --ci --require-executed
Bounded FP32 adapters and weight preparation
attach_lora_adapters accepts explicit target, rank, alpha and name declarations for multiple unique biasless FP32 projections before backward synthesis. It reuses ordinary projection, scale and add operations and the existing generated training backend. A declared parameter budget is checked against resolved ranks; the authoring utility does not guess targets or reduce ranks silently. Tied/shared targets, unsupported dtypes and inconsistent geometry fail.
Each inserted base projection has an adapter-qualified operation identity and a separate projection semantic label. The original operation identity stays on the join that produces its former output. Name collisions fail before lowering. A two-adapter composition passes the shared provenance audit and independent update comparison.
resolve_lora_parameters returns names, shapes, layout, dtype and trainability without parameter values. prepare_lora_weights validates every base parameter, initializes wholly absent adapters with explicit seeds, and returns schema/specification/weight hashes plus initialization method, implementation and NumPy version. Existing adapters are preserved; partial adapters, stale scale values and unused initialization seeds fail. Resume requires all parameters and performs no initialization. Optimizer moments and counters still resume through the existing checkpoint interface.
from version.v8.training.lora import attach_lora_adapters
from version.v8.training.lora_weights import prepare_lora_weights
adapted, specs = attach_lora_adapters(authored_training_ir, adapters=[
dict(target="projection.weight", name="task", rank=8, alpha=16),
])
weights, evidence = prepare_lora_weights(
adapted, base_weights, seeds={"task": 41})
# Compile the adapted graph through the existing training lowering/backend.
# Pass these checked weights to its native loader; Python does not update them.
resumed, resume_evidence = prepare_lora_weights(
adapted, checkpoint_weights, mode="resume")
This is supported training-IR authoring, not arbitrary PyTorch module conversion or an all-model notebook API. The default initialization remains PCG64 normal A with standard deviation 1/sqrt(input width), zero B and frozen FP32 scale. Initialization is artifact preparation outside native computation. Generated C executes forward, gradients and AdamW updates.
The asset-free LoRA suite compares four nonzero native training compositions with independent PyTorch across 40 updates, including two adapters, rectangular projections, frozen-base gradient flow, evaluation, checkpoint resume and inference export. It also rejects malformed schemas and partial initialization. A shared-builder portability test runs two adapters for 40 updates from a C-only host, moves the bundle, removes the build directory, and reproduces identical results. Frozen base matrices and scales remain unchanged; removing the bundled engine fails. This does not certify BF16 adapters, quantized-base backward, QLoRA or complete connected Clef training.
CK_NUM_THREADS=4 OMP_NUM_THREADS=1 python -m pytest -q tests/test_v8_lora_training.py python scripts/nightly_runner.py --tests training_lora_fp32 --json /tmp/lora.json
Map-owned projection and addition differentiation
FP32 checked row projections (with or without bias) and row addition declare
training_contract.differentiation in their kernel maps. The shared
circuit adapter resolves the operation interface, ready gradient rule, saved
operand ports and dimension aliases from that declaration. A provider name or
prefix no longer establishes training support for these operations.
The bounded cke.training.direct-vjp.v1 contract reuses the same
forward provider and existing operand tensors. It validates FP32 ports, saved
operands and positive bounded dimensions before backward generation. Ready rules must explicitly admit the operation interface and provide exactly the declared input/weight gradient destinations with compatible ownership. Missing, unrelated, duplicate or misowned destinations are rejected even when saved operands match. It does not
introduce a new derivative executor or an expression language. Cached-forward
rewrites and other operation groups still use explicit translation rules; this
is not arbitrary-circuit differentiation or BF16 adapter certification.
Generated nonzero projection compositions, complete supplied-feature head training and mixed-projection regressions retain independent PyTorch checks. Unavailable pinned assets remain SKIP. Connected backbone/head numerical acceptance remains separate and unresolved.
A released-size Ryzen replay compares all 121,762,820 gradient elements and 123 exported tensors exactly with the retained baseline. A relocated C-only host reproduces its update. This is one supplied-feature head update, not a fresh backbone oracle run; the prior connected score FAIL remains visible in the identity-bound evidence.
python -m pytest -q tests/test_v8_map_training_binding.py tests/test_v8_circuit_training_lowering.py tests/test_v8_clef_evaluation.py tests/test_v8_clef_full_training.py tests/test_v8_clef_mixed_training.py
Composition and training audit follow-up
The composition gate now requires the declared consumer operation to contribute to the requested output. A live sibling cannot make a dead declared consumer pass. Boundary identities, shape, dtype and element count must be present and consistent. Structural, executed and numerical verdicts remain separate.
Each head update retains exact frozen-embedding verification through a generated native tensor comparison. It does not export all parameters for this check. Full snapshots remain available for certification and export. On the released 121,762,820-element head, one Ryzen update avoided a 488,296,464-byte host weight snapshot and compared 1,245,184 bytes in place. The native update took 14.153 seconds and its comparison 0.239 ms in that run; these are not whole-job throughput claims.
Broadcast expansion uses declared immutable int32 constants owned by the generated runtime,
not a host input named broadcast_zeros. An unrelated broadcast-plus-GELU graph passes
40 PyTorch gradient/AdamW comparisons, including a trainable broadcast row. Maximum gradient,
updated-weight and moment errors were approximately 1.27e-7, 1.07e-6 and 4.47e-8 under the existing
bounds. Invalid constants and attempts to write them are rejected.
The released-head update and gradients exactly replay the retained independent-oracle baseline; a relocated C-only host also reproduces the update. This is supplied-feature head training, not a fresh connected backbone oracle run. The prior connected score comparison remains FAIL, maximum error 0.009174 at a 2e-6 bound. Evidence and identities.
python -m pytest -q tests/test_v8_generated_composition_contract.py tests/test_v8_circuit_training_lowering.py tests/test_v8_clef_evaluation.py python scripts/nightly_runner.py --tests generated_composition_training --json /tmp/composition-training.json
Relocatable native head training
The head recipe uses the existing shared training build helper and packages
libckernel_engine.so beside train.so, with a
$ORIGIN loader path. Build identity records the compiler, generated
source, extra provider sources and engine hash. Additional neural sources are
resolved from generated provider declarations and kernel maps when their symbols
are absent from the engine; the head no longer names a special source file.
Missing source contracts fail before compilation. Resume validates the packaged
engine rather than consulting the checkout's build directory; older runtimes
without that dependency must be regenerated.
A C-only regression host runs generated forward, backward and AdamW after the bundle is moved outside its build directory. The model-neutral graph completes 40 updates with frozen weights unchanged. A complete small Clef head reproduces three native updates and evaluation exactly after relocation. Removing the packaged engine causes startup failure. No Python interpreter or repository headers are required during those deployed runs.
Released-head relocation evidence: On Ryzen, the full 121,762,820-element trainable head completed one AdamW update through a relocated C-only host. Loss changed from 0.0080747874 to 0.0078037283; all updated weights matched the native recipe exactly. Current gradients and updated head tensors also match the retained independent PyTorch-certified baseline exactly. This was a baseline replay, not a fresh oracle run. The single process took 18.22 seconds including initialization and evaluations; it is not a performance benchmark. Artifact identities and separate verdicts retain the connected-model numerical FAIL.
The private full-size reproduction is retained on Ryzen at
/data/cke/private/clef-portable-head-20261114/reproduce.py;
run it with the pinned oracle environment and a new output directory argument.
It consumes the retained native feature capture and released head, so its scope
is explicitly head-only. The small asset-free regression remains in ordinary
nightly; it does not impersonate this full-size run.
This is runtime portability on the tested Linux ABI, not arbitrary-machine binary portability or released 9B connected-model certification. The test host loads prepared parameters and feature requests; text preprocessing and a declared generated backbone-to-head connection remain separate work. The ordinary recipe's Python layer still manages datasets and reports.
The DSL policy now covers the shared circuit-training adapter, composition gate, checked emitter and bundle compiler, plus shared backward synthesis, training memory planning, code generation and runtime building. Clef and Kokoro are explicitly included in forbidden model-dispatch literals. This checks those policy rules; it is not proof that all compiler behavior originates in IR.
Declared native feature connection: development evidence
A model-neutral native sequence provider resets an exclusively owned decoder, executes its token entry, stages checked FP32 feature rows, resets state, and publishes only after success. Circuit authoring connects those rows to an existing head through ordinary edges. The host supplies inputs and a borrowed component descriptor; it does not run the token loop or copy intermediate feature rows.
The synthetic decoder plus complete small head executes in a relocated C-only process after the build directory is removed. Poisoned hidden features and logits are replaced by native output; missing component libraries fail. Failure and cancellation tests preserve unpublished output and permit recovery. Borrowed descriptors hold exclusive decoder ownership; reentrant encode, close or borrowing is rejected, and failed initialization releases native state.
Released-model execution: one pinned 128-token request now invokes the 9B generated decoder component and complete head through one generated entry, with no Python token loop or captured hidden input. Invalid IDs are rejected and a repeated request succeeds. A separate C-only process outside the checkout reproduces current logits exactly. A repeated C-only run records the actually loaded shared-library hashes and thread environment; neither libpython nor libtorch appears in its loaded-library map. Internal backend dispatch choices remain unobserved. This is a fixed diagnostic artifact, not a persistent arbitrary-text service or connected backward graph.
A fresh pinned PyTorch head comparison on the same current native features produces exactly matching option logits (maximum absolute error 0, unchanged absolute bound 2e-6). This isolates head arithmetic; it does not certify the backbone trajectory.
Numerical limitation: current Python-driven and native token loops agree exactly, but both differ from the older native capture beginning at token index 54. That retained comparison remains FAIL; its cause is unresolved. The independent connected PyTorch failure also remains open. The execution identities and separate verdicts do not convert either discrepancy into a pass. The provider declares no backward support; supplied-feature head training remains a distinct path.
The map declares its call ABI and numerical copy contract. Native-entry parameter types must exactly match the provider declaration; code generation rechecks the binding and rejects altered pointer expressions. Component arithmetic, artifact identity and lifetime remain separate obligations; an exact feature copy does not certify the decoder that produced it.
python -m pytest -q tests/test_v8_native_entry_binding.py tests/test_v8_decoder_feature_sequence.py tests/test_v8_native_feature_head_composition.py python scripts/nightly_runner.py --tests decision_native_frontend --json /tmp/native-feature-boundary.json
Remaining before deployment: expose the verified composed artifacts through a stable package/loader interface, certify connected outputs, and provide a checked native-context input for frozen component execution in training. The released C-only trial uses existing model artifacts outside the checkout; it does not certify arbitrary-machine portability. Runtime configuration must declare the separate backbone-token input as int32; compact head embedding IDs are not interchangeable with backbone vocabulary IDs.
Model topology lives in the circuit builders here. Kernels implement arithmetic; maps describe ABI, scratch and numerical contracts. Generic operation catalogue entries do not inspect the model name.
Native text → backbone → decision → head training
The offline diagnostic now runs the released Clef-Flash Qwen3.5-9B backbone through ordinary generated decode execution, reads its declared final feature row, and passes those CKE-produced features and pinned output-embedding rows into the generated head. No PyTorch or Transformers import is needed for production inference or head training. Python owns schema encoding, tokenization, artifact preparation, requests and reporting; generated C owns neural forward/backward and AdamW.
Generated composition: NOT_IMPLEMENTED. End-to-end certification: NOT_TESTED.
The host currently extracts backbone features, prepares embedding rows and builds a separate
head package. Two native components connected by Python do not satisfy the generated
backbone-to-head composition gate. This is component integration evidence, not a complete
compiled inference/training graph. The CLI option --require-generated-e2e
fails before execution on this diagnostic path.
The shared bounded circuit-training adapter now owns operation-to-training translation; Clef authoring supplies its graph, inputs, frozen parameters and grouped loss. Cosine VJP semantics are explicit circuit metadata, never inferred from operation names. The adapter still has a bounded provider inventory; it does not imply arbitrary inference circuits support training. A non-Clef graph exercises generated forward, backward, frozen-weight gradient propagation, evaluation and 40 native AdamW updates against PyTorch. Standard nightly checks reject missing composition evidence and host-fed internal edges. Structural checks remain distinct from execution, gradient and numerical certification.
Feature batches now use the shared generated FP32 input ABI:
ck_train_set_float_inputs_ex. Hidden features occupy input storage,
not a frozen __features.hidden parameter. Batch binding copies only
feature data and does not export/import the parameter array. Counts, alignment,
overlap and finite values are checked before publication; rejection invalidates
input readiness and leaves parameters and optimizer state unchanged. Backward
cannot reuse a forward pass after the feature binding changes. These inputs are
nondifferentiable external features; this is not a substitute for a connected
backbone activation edge when backbone training is requested.
Checkpoints retain parameters and optimizer state, not feature batches. Inputs must be supplied again after initialization/reload. Existing generated runtimes without this ABI must be regenerated; their old runtime identities are not silently accepted as compatible. This remains a fixed-capacity input contract.
The backbone is frozen. The 121,762,820-element released head is trained using
train_clef_head.Runtime, exported and reloaded through generated inference.
This does not train the backbone, introduce LoRA, or establish held-out task accuracy.
The head is compiled for each offline request geometry; a persistent arbitrary-text
application endpoint remains a separate integration gate.
The Qwen3.5 circuit now selects existing BF16-storage providers for recurrent softplus, attention gating and text rotary arithmetic when BF16 recurrent projection weights are declared. Identical-input replay isolated these contract mismatches: FP32 gating differed by 0.00481534, and position-one rotary Q differed by 0.01765776. The existing BF16 providers matched the corresponding PyTorch values exactly. FP32 and quantized selections remain unchanged. The recurrent softplus replay also becomes exact (previous maximum difference 3.55e-9). These are circuit contract fixes, not new model-specific compiler branches.
The diagnostic profile explicitly selects the existing oneDNN 3.12 projection contract. Its reference is Transformers 5.10.2, the release's recorded version, with SDPA attention. An experimental Transformers checkout and eager attention are retained as diagnostic comparisons, not silently treated as the same oracle.
Released-weight evidence, 2026-11-14
On Ryzen 9950X3D, one 128-token sentiment request ran through the actual released backbone and head with PyTorch/Transformers imports blocked. One native AdamW update reduced this record's loss from 0.008074787 to 0.007803728; exported/reloaded logits were exact. This is a training-mechanics test, not a held-out accuracy result.
| Independent check | Maximum absolute error | Verdict |
|---|---|---|
| Head on identical native features | 9.53674e-7 (bound 2e-6) | PASS |
| All 121,762,820 head parameter gradients | 1.22469e-7 (bound 3e-5) | PASS |
| AdamW-updated weights | 2.38419e-7 (bound 3e-6) | PASS |
| Connected logits versus SDPA full prefill | 0.00803661 (bound 2e-6) | FAIL |
| Connected logits versus SDPA incremental execution | 0.00917447 (bound 2e-6) | FAIL |
The native path executes end to end, but is not numerically certified. The three exact local fixes do not resolve the complete backbone trajectory. The first token's final features are exact; the second token still diverges. Continue identical-input state/operation diagnosis before expanding LoRA. No tolerance was increased and no reference intermediate entered production.
A further state capture identifies a precision-policy difference: the released Transformers incremental cache stores BF16, while CKE's selected recurrent contract retains FP32 state. Diagnostic BF16 rounding of CKE's first-step state reproduces all 524,288 reference values exactly. A separate reference experiment retaining FP32 cache state still has connected logit error 0.00187111 and remains FAIL. This isolates one cause, not all remaining divergence. Production state precision is unchanged; the modified-reference experiment is explicitly labelled diagnostic, not released parity.
Machine-readable evidence retains raw logits, RMSE, reference signal magnitudes, identities, dependency hashes, training results and local replay results. Private activation packets remain on the asset host; the public artifact contains the synthetic request and summary metrics.
Reproduction and boundaries
Use the pinned release fb4417d31c1d0f9e7e77f1141b40aaede7491e00,
its SHA-verified release lock, and standalone native oneDNN/SLEEF/MKL dependencies.
This profile is validated on AVX-512-capable Ryzen; it is not an AVX2-only portability claim.
Set CK_ONEDNN_LIBRARY, CK_SLEEF_LIBRARY and
CK_MKL_LIBRARY to those installed libraries. Record their hashes and thread
settings. The build command requires a fresh output directory; it never reuses unbound
converted weights. A local trusted administrator must supply the expected lock and
bundle hashes.
python3 version/v8/decision/build_native_clef_backbone.py \
--release "$RELEASE" --lock "$LOCK" --lock-sha256 "$LOCK_SHA" --output "$BUNDLE"
python3 version/v8/decision/run_native_clef.py \
--release "$RELEASE" --lock "$LOCK" --lock-sha256 "$LOCK_SHA" \
--bundle "$BUNDLE" --bundle-descriptor "$BUNDLE/bundle.json" \
--bundle-sha256 "$BUNDLE_SHA" --request request.json --output native-run
python3 version/v8/decision/train_native_clef.py \
--release "$RELEASE" --native-run native-run --report-sha256 "$REPORT_SHA" \
--labels labels.json --output training-run --steps 1 --workspace-mib 4096
python3 tests/run_clef_native_oracle.py \
--release "$RELEASE" --native-run native-run --native-sha256 "$REPORT_SHA" \
--capture "$CAPTURE" --capture-sha256 "$CAPTURE_SHA" --output comparison.json
python3 tests/run_clef_native_training_oracle.py \
--release "$RELEASE" --native-run native-run --native-sha256 "$REPORT_SHA" \
--training training-run --training-sha256 "$TRAINING_SHA" \
--labels labels.json --output gradient-comparison.json
The independent capture tool accepts --attention sdpa --schedule incremental
to match the native token-by-token schedule. Its defaults remain eager/prefill for
existing users. Full-prefill and incremental comparisons are retained separately;
changing the oracle schedule is not permission to erase a failing comparison. Neither reference features nor reference logits enter
the native runner. Reports separate execution, same-input head parity, connected
logit parity, exact backbone-storage agreement, gradients, optimizer updates and reload.
Unavailable reference evidence is NOT_TESTED, not PASS.
The released geometry needs approximately 2.51 GB of native training workspace. The offline CLI explicitly permits a 4 GiB budget; existing application recipes retain their 2 GiB default. This is an arena bound, not total process memory: weights, compiler, Python arrays and exported snapshots require additional RAM. Evaluation performs no optimizer update. Offline tools do not claim HTTP deadlines, cancellation, concurrent backbone ownership, streaming training datasets or automatic application admission.
Cheap standard-nightly coverage checks schema spans, rejection without truncation,
BF16 versus FP32/quantized contract selection, artifact tampering, decoder ownership,
short/long/short state reset, output lifetime, partial failure and recovery, and a
nonzero generated head against an independent reference. Full 9B/released-head runs
require the pinned assets and are reported separately. Set
CKE_CLEF_NATIVE_ACCEPTANCE to a trusted JSON file with the runner arguments
release, lock, lock_sha, bundle, bundle_descriptor, bundle_sha, request
and a labels mapping (question ID to candidate ID) to enable the released
asset execution/training/reload test. Without that configuration it reports SKIP.
That test is execution evidence; run the independent comparison commands separately.
Validation loss correction: forward plus declared loss, without an update
A real notebook consumer at aa68f5812 exposed stale held-out metrics.
Runtime.evaluate() ran forward and read ck_train_get_loss(),
but that accessor returns cached loss. The grouped loss seed executes at the start
of backward, so forward alone returned zero after initialization and the last
training loss after updates. Native training, exported weights and scoring were
correct; the reported validation metrics were not.
The generic generated ck_train_eval_step(float *loss_out) now executes
forward plus the single declared loss-seed operation. It does not run parameter
backward or an optimizer step. The existing loss provider refreshes its output-gradient
seed and scratch as well as the scalar loss; forward activations and supplied feature
inputs also change. Trainable weights, parameter gradients, AdamW moments and update/
accumulation counters remain unchanged. A failed evaluation does not publish a scalar.
Ordinary decoder cross-entropy and grouped question loss use the same contract;
there is no Clef-specific compiler branch or Python loss calculation in production.
| 40-update notebook reproduction | Old report | Corrected native report | Independent grouped CE |
|---|---|---|---|
| Validation before training | 0.0 | 0.2278986424 | 0.2278986458 |
| Validation after training | 0.1184638888 | 0.1780027747 | 0.1780027814 |
The reproduction uses the original pinned synthetic package, 20 epochs, learning rate 0.001 and 40 native updates, with PyTorch/Transformers imports blocked in the production process. All 40 training losses, trained weights, optimizer-state hash and record logits are unchanged. The original notebook evidence is preserved. Retained numerical evidence and artifact identities separates the old and corrected reports.
Regressions cover initial evaluation, different records and labels, unequal option counts, validation between updates, rejected evaluation and recovery, independent FP64 stable log-sum-exp, multiple held-out records, checkpoint reload, final report metrics and published job metrics. They separately preserve inference export/reload checks. Validation-before means the state at the start of this invocation; when resuming, it is evaluated at the loaded checkpoint, not at the original initialization.
The generated runtime contract now includes evaluation semantics. Regenerate old training libraries; a cached runtime without the new entry rejects rather than silently using stale loss. Existing reports must be regenerated, not relabeled. Numerical bounds are unchanged, and this fix does not turn the mixed-BF16 trajectory into a pass or establish task quality.
python -m unittest tests.test_v8_clef_evaluation tests.test_v8_grouped_loss_training tests.test_v8_selective_training -v
python scripts/nightly_runner.py --tests training_clef_registered_recipe --require-executed
Training bridge v2: opaque revisions and pinned worker identity
cke.training.jobs.v2 adds a bounded local contract for external application
backends. It preserves the application's 32-character lowercase hexadecimal
client_revision verbatim in the reviewed plan and durable job. It does not
cast or hash that revision into an integer. Existing v1 integer-revision clients
continue to use their unchanged contract. V2 plans/jobs carry a null legacy
draft_revision and explicitly versioned schemas.
This milestone is trusted local integration only. It opens no socket, adds no remote authentication, and does not reuse read-only control discovery for training execution. The principal comes from the trusted supervisor, never JSON. A future authenticated transport must bind a verified backend credential to the appropriate tenant/owner; a caller-selected owner or a tunnel is not authorization.
Lifecycle and freshness
- Discover approved recipes and registered package IDs/hashes, geometry and split
counts. Retain the returned
worker_boot_idandcapabilities_sha256. The latter hashes the complete v2 capability body, excluding its own digest. Hashes detect changed content; they do not authenticate a peer. - Every non-discovery request includes both identities and the exact client revision. Preparation validates the configuration and persists a reviewable plan without executing training. The plan hash binds owner, configuration, revision, backend capabilities and expiration.
- Submit the reviewed
plan_sha256with a stable owner-scopedidempotency_key. An uncertain timeout must retry the same plan/key/revision; it must not invent a new key. An exact duplicate returns the existing receipt. - Observe and retrieve artifacts only for the authenticated supervisor's owner and matching revision. A different application draft must not publish an older job as current. The independent client rejects mismatched request hashes, boot/capability identities, revisions and corrupted artifact streams.
- After worker restart, discover its new boot identity before observing or retrying.
An existing idempotency receipt can be recovered, but ambiguous interrupted jobs stay
recovery_required; they are never automatically relaunched.
Revisions are opaque, not ordered counters. CKE binds them to the submitted plan; the application remains responsible for tracking which draft is current. Unknown or unavailable capabilities remain unavailable. Missing metrics are absent, never fabricated as zero; clients should display them as unavailable. Cancellation remains unsupported.
Local setup and wire examples
Use the existing administrator package registry and private job store described below. The principal argument is supervisor configuration, not a field accepted from clients. The following command performs discovery only and contains no credentials:
printf '%s\n' '{"schema":"cke.training.jobs.v2","request_id":"discover-1","operation":"discover"}' | \
python -m server.training.jobs --store /private/jobs --principal demo \
--clef-registry /private/registry.json
A successful reply uses these fields (placeholders below are explanatory, not a valid pinned response):
{"schema":"cke.training.jobs.v2","request_id":"discover-1",
"request_sha256":"SHA256_OF_EXACT_REQUEST","worker_boot_id":"BOOT_HEX_32",
"capabilities_sha256":"CAPABILITIES_HEX_64","ok":true,"error":null,
"result":{"schema":"cke.training.jobs.v2","worker_boot_id":"BOOT_HEX_32",
"capabilities_sha256":"CAPABILITIES_HEX_64",
"operations":["discover","prepare","submit","observe","artifact"],
"cancel":"unsupported",
"transport":"trusted_local_only; remote authentication NOT_IMPLEMENTED"}}
The actual discovery result additionally contains the complete registered validation capabilities, execution scope, limits and authorization description. Never hash or pin this abbreviated example. Prepare with the actual returned identities:
{"schema":"cke.training.jobs.v2","request_id":"prepare-1","operation":"prepare",
"worker_boot_id":"BOOT_HEX_32","capabilities_sha256":"CAPABILITIES_HEX_64",
"client_revision":"00ffffffffffffffffffffffffffffff","experiment_id":"decision-demo",
"config":{"recipe":"clef_head_fp32_v1","package_id":"demo",
"package_sha256":"REGISTERED_PACKAGE_SHA256","dtype":"fp32","optimizer":"adamw",
"method":"head_only","epochs":2,"learning_rate":0.001}}
Submit requires plan_sha256 and idempotency_key; observe requires
job_id; artifact requires job_id, artifact_id and
offset. All include schema, request ID, operation, boot/capability identities
and client revision. Unknown fields, client paths, principals and shell commands reject.
Artifacts retain the existing 8192-byte chunk limit and manifest SHA-256 verification;
an application must retain the expected manifest and verify the entire resumed transfer.
Evidence and remaining agreement
Asset-free independent consumer tests cover no-execution preparation, exact revision preservation, duplicate/lost-reply recovery, restart, stale identities, cross-owner denial, wrong revisions and corrupt/truncated artifact responses. These tests simulate execution; they do not certify native arithmetic or the actual laptop application. The separate opt-in native Clef job test exercises opaque revisions through training and export.
Before exposing a remote service, agree and test authenticated ZeroMQ peer/tenant mapping, server identity pinning, bounded messages and deadlines, disconnect/reconnect behavior and transport error envelopes. V2 local error handling remains the existing stdio rejection envelope. Do not advertise a production network endpoint or cancellation until those contracts exist. No backward kernels, optimizer math or compiler lowering change in this bridge milestone.
python -m unittest server.tests.test_training_bridge server.tests.test_training_jobs -v
python scripts/nightly_runner.py --tests training_job_lifecycle --require-executed
CKE_RUN_CLEF_JOB=1 python -m unittest \
tests.test_v8_clef_training_recipe.NativeRecipe.test_opaque_revision_native_job_and_export -v
Application training: larger supplied-feature packages
The FP32 head recipe now admits up to 4,096 records and 512 token positions, with 1–1,000 epochs and a combined ceiling of 100,000 updates. The loader retains its 512 MiB decoded-dataset bound and the compiler retains its 2 GiB native-workspace bound. These are admission ceilings, not evidence that every maximum-size geometry fits memory or has acceptable speed. Package manifests are bounded to 8 MiB.
Application clients must rediscover capabilities: clef_head_fp32_v1.limits
now explicitly reports records, tokens, update budget, dataset bytes and the exact-length
restriction. The independent simulated consumer and job-progress checks use the expanded
contract. An administrator can construct Jobs(..., wall_seconds=3600)
for a longer local run; the default remains 600 seconds and the accepted range is
60–86,400 seconds. This is local policy, not a client-requested timeout override.
Cancellation and remote authentication remain unsupported.
The trainer indexes the schedule instead of materializing one record reference for every update. Every 1,000 updates before the final/interrupted step it evaluates validation records and publishes an identity-bound checkpoint. Only the last two periodic checkpoints created by that invocation are retained; a resumed invocation preserves earlier files. The final checkpoint and explicit stop-after behavior remain available. Resumption requires the same dataset and configuration. Older checkpoint configurations without the new checkpoint/evaluation policy are not silently accepted.
The final report includes per-split, per-question accuracy, macro-F1, support and confusion matrices, plus record logits, labels and stable per-question softmax scores. Candidate identities, not their array positions, define classes. Question IDs must denote the same task across records. Macro-F1 includes declared classes with zero support (zero-division score zero); softmax scores are not calibrated confidence. Sparse confusion entries are always supplied; dense matrices are omitted once their combined 65,536-cell budget is exhausted. This avoids quadratic reporting memory for many candidate identities. Reports include elapsed time and process high-water RSS on Linux. Timing includes preparation/export and is not a kernel-throughput measurement.
Serving fixtures no longer grow with the training dataset: export registers at most
64 question examples, prioritizing the first validation record then dataset order, and explicitly reports
inference_export_record_ids. Native training-runtime evaluation still reports
all records. Independent generated-inference export checks apply to the selected subset;
the exported head contains the trained weights, not a claim of corpus-wide export parity.
Executed stress test: 34 synthetic records, 96 token positions, 41 epochs / 1,353 updates, periodic validation at update 1,000, and 32 exported two-question records. This tests limits and lifecycle behavior, not sentiment accuracy or Qwen execution. The existing distinct-record pinned PyTorch head-update comparison also passes without tolerance changes.
make -j4 build/libckernel_engine.so python -m unittest tests.test_v8_clef_dataset_scale -v python scripts/nightly_runner.py --tests training_clef_registered_recipe --require-executed python version/v8/decision/train_clef_head.py \ --package /absolute/path/package.json --package-sha256 EXPECTED_SHA256 \ --run-dir /absolute/path/new-run --json-out /absolute/path/result.json \ --epochs 50 --lr 0.001
Still required before claiming general real-text training: each package has one exact token/question/option geometry. No implicit padding or attention masking is provided. Mixed-length capture, bucketed shared-optimizer training, a checked multi-record capture-to-package converter, automatic best-validation-checkpoint selection, and a separate final test-set workflow remain follow-ups. The application should preserve train/validation/test separation and must not select models using the final test set. Native backbone execution, full-model LoRA and QLoRA are not enabled by this expansion. No real dataset benchmark or application-quality claim has been made.
Registered FP32 Clef head training
This opt-in recipe trains the complete supported Clef head on supplied backbone
features. It does not execute or train a backbone, tokenize text, add adapters,
or execute application actions. A successful job means generated training,
frozen embedding checks, checkpoint reload and generated inference export passed.
It does not certify the submitted dataset's quality, calibrated confidence or
numerical parity against an oracle. Those report fields remain NOT_TESTED.
Ownership and precision
The existing v8 circuit authoring, backward rules and shared training code generator produce native forward, backward and AdamW execution. Python validates records, loads input features, invokes complete generated steps and packages artifacts. It never computes model gradients or updates parameters. No new compiler branch, model scheduler or training stack is introduced.
The admitted recipe is clef_head_fp32_v1: FP32, AdamW, per-question cross entropy,
1–1,000 epochs (at most 100,000 updates), learning rate (0, 0.01], epsilon 1e-4, betas (0.9, 0.999), weight
decay 0.01, no dropout, gradient accumulation 1. All head parameters train;
output embedding rows and supplied backbone features are frozen. Mixed BF16,
LoRA and QLoRA are unavailable through this recipe. In particular, the existing
mixed-BF16 trajectory failure has not become a pass.
Dataset/package contract
cke.clef.training-package.v1 has exactly five fields:
schema: the version above.model:id,revision,backbone_id,backbone_revision,tokenizer_revision, andfeature_provenance: "supplied". Revisions are pinned hexadecimal identities. Supplied provenance records a declaration, not proof that CKE executed that backbone.geometry:tokens,channels,width,heads,feedforward,vocabulary,questions,options. All records share this declared geometry.weights: local basenamefileandsha256of the FP32 NPZ raw head weights, includingoutput_embedding.weightand packed QKV weights.records: 2–4,096 records withid(at most 60 characters),split(trainorvalidation),asset(file,sha256),question_idsand groupedcandidate_ids. Both splits must be present. IDs preserve question/option order.
Each record NPZ contains hidden (FP32) and int32 input_ids, question_spans,
option_spans, global_span, option_counts, question_type_ids, labels.
Spans are half-open. Global span must select the last valid token. Labels index
that question's allowed options; option counts partition flattened option rows.
Arrays, finite values, IDs, counts, shapes, archive expansion and all asset hashes
are checked before code generation. Archives must not contain Python objects.
See version/v8/decision/training_package.py for exact bounds and inventories.
The tests create synthetic, nonzero examples with overlapping spans and two
questions with unequal option counts. Do not use private customer data for the
first client demonstration.
Run locally
Build build/libckernel_engine.so first. Use a trusted package and retain its hash:
python version/v8/decision/train_clef_head.py \
--package /absolute/path/package.json --package-sha256 SHA256 \
--run-dir /private/new-run --json-out /private/result.json --epochs 2
The output contains native runtime/source identities, held-out loss before/after,
per-record raw logits and identities, an identity-bound optimizer checkpoint,
trained_head.npz, and serving.zip. The latter contains the ordinary generated
head package and registered supplied feature artifacts. The recipe loads it with
NativeHead and compares its logits with the trained runtime for every record.
For a deliberate CLI checkpoint, add --stop-after 2; continue with the same
package, configuration and run directory using --resume /private/new-run/checkpoint-2.
This restores weights, AdamW state and step count. The saved runtime and engine
identities must match. Completed checkpoints cannot be resumed as new training.
This is local CLI resume, not a job-service resume or cancellation promise.
Independent client and job service
The existing cke.training.v1 discovery/validation API remains non-executing.
An administrator can opt into the head recipe by registering a package mapping:
{"demo":{"path":"/absolute/path/package.json","sha256":"SHA256"}}
Launch trusted local stdio, from a clean committed source tree:
python -m server.training.jobs --store /private/jobs --principal demo \
--clef-registry /private/registry.json
Use discovery before constructing a request. A head configuration is:
{"recipe":"clef_head_fp32_v1","package_id":"demo","package_sha256":"SHA256",
"dtype":"fp32","optimizer":"adamw","method":"head_only",
"epochs":2,"learning_rate":0.001}
prepare binds configuration, worker boot, capability/source/interpreter/package
identities and expiration. submit returns a durable acknowledgement with an
idempotency key. Only one job executes, with no backlog; discovery and preparation
do not execute training. observe reports durable monotonic progress and metrics.
Head metrics use step_kind: update and unit: nats/question; they must not be
interpreted as decoder token losses. Artifact retrieval uses opaque IDs, bounded
chunks and hashes. The independent clients in server/training/consumer.py and
job_consumer.py do not import the worker implementation.
Readiness and gaps
| Boundary | Status |
|---|---|
| Supplied-feature FP32 generated head training and inference export | Implemented; synthetic native regression |
| Independent small-head PyTorch comparison | Optional pinned-source test; explicit skip when unavailable |
| Frozen output embeddings and fresh-process CLI resume | Native regression |
| Held-out loss | Reported; no automatic quality acceptance |
| Backbone execution, text-to-decision | Not provided by this recipe |
| Job cancellation / automatic restart / remote resume | Unsupported; interrupted jobs become recovery_required |
| Remote authenticated transport | Unsupported; trusted local stdio only |
| BF16 trajectory certification, LoRA, QLoRA | Not admitted |
Do not expose this interface as an authenticated remote service. A caller-provided
principal is meaningful only inside a trusted local supervisor. A tunnel is a
development bridge, not authorization. Evaluation continues to use the separate
cke.decision.evaluate.v1 contract; neither evaluation nor control discovery is
a training command. Keep the current decision HTTP worker loopback-only.
The service exports checkpoint metadata and serving artifacts, but remote
checkpoint-state transfer/resume is not yet an application contract. Applications
must surface recovery_required, NOT_TESTED and unsupported operations explicitly.
Validation retained for this change
On the local Intel i7-14700T host with PyTorch 2.12.1+cpu and one requested
OpenMP thread:
- Focused recipe and existing service/client suites: 31 passed, 2 explicit skips (opt-in real SVG and Clef jobs).
- Real registered Clef job rerun separately from the clean committed tree: passed, including independent observation and chunked artifact hash verification.
training_clef_registered_recipethrough the standard nightly runner: passed.- Existing independent FP32 complete-head 40-update trajectory: passed. Loss
0.28329045 → 0.06633431; max absolute gradient error
1.374632e-6, weight error1.668930e-6, optimizer-moment error3.147870e-7. Checkpoint/resume and generated inference export passed. These features and head geometry are synthetic; this run does not certify released-geometry training or task quality.
Reproduce the new boundary with:
OMP_NUM_THREADS=1 CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py \
python -m unittest tests.test_v8_clef_training_recipe \
server.tests.test_training_contract server.tests.test_training_jobs -v
OMP_NUM_THREADS=1 CKE_RUN_CLEF_JOB=1 python -m unittest \
tests.test_v8_clef_training_recipe.NativeRecipe.test_registered_job_and_independent_artifact_client -v
python scripts/nightly_runner.py --tests training_clef_registered_recipe --require-executed
The oracle source must match the SHA-256 in
tests/fixtures/decision/clef_flash/identity.json; no source means explicit skip.
The production resume test blocks imports of torch and transformers.
FP32 memory-projection boundary: useful diagnosis, still not certified
The bounded --mixed --fp32-memory-projection experiment retains the existing FP32
memory projection in the authored training graph. Other selected projections retain BF16 operands.
The independent PyTorch reference uses its original FP32 linear operation for that same projection.
The report lists actual selected providers and the exception. There is no production kernel, map,
compiler or application admission change.
All five experiments were rerun on the same build of main f6f5e6b1e, seed 611,
CPU PyTorch 2.12.1. The complete small head still uses synthetic frozen features and a bounded task;
these are not generalization or performance measurements.
| 40-update comparison | Worst gradient error | Verdict |
|---|---|---|
| FP32 baseline | 1.375e-6 | PASS |
| All selected projections mixed, same state | 6.748e-4 | FAIL at update 18 |
| FP32 memory exception, same state | 4.170e-5 | FAIL at update 37 |
| All selected projections mixed, independent trajectory | 0.0328365 | FAIL from update 8 |
| FP32 memory exception, independent trajectory | 0.00409246 | FAIL, first at update 21 |
The exception removes the earlier memory-projection rounding failure, but a new same-state
rounding boundary appears at evidence1_feedforward_norm_op → evidence1_feedforward1.
Its first affected input is -1.4179686307907104 versus -1.41796875, rounding to -1.4140625 versus
-1.421875. Identical-input projection replay gives zero output error and dW error 5.821e-11.
The independent trajectory additionally encounters a QKV master-weight rounding boundary.
This is evidence against treating one FP32 exception as a general solution.
Thresholds are unchanged. Both mixed policies retain numerical FAIL. Native state hashes match
across same-state/independent diagnostic modes for each policy; resume remains exact. Released-weight
mixed precision, mixed inference export and performance remain NOT_TESTED. Full reports and packets
are referenced in docs/notes/clef_fp32_memory_boundary_20261113.json.
python tests/run_clef_training_oracle.py --mixed --fp32-memory-projection --same-state \
--report /tmp/clef-boundary/same-state.json
python tests/run_clef_training_oracle.py --mixed --fp32-memory-projection \
--report /tmp/clef-boundary/independent.json
Application-agent readiness
The numerical experiment above is separate from the
registered FP32 head-training recipe.
With an administrator-registered supplied-feature package, the local job service
now admits clef_head_fp32_v1, executes generated forward/backward and AdamW,
and exports a native inference package. Clients must discover the running worker's
capabilities before submitting; clef_head_v8 is not a supported alias.
This is head-only integration, not raw-text/backbone execution. BF16 certification, LoRA, QLoRA, remote authentication and job cancellation remain unavailable. The mixed-policy numerical failures above remain unchanged.
Same-state update diagnosis: 39 of 40 comparisons pass
--same-state copies CKE's pre-step master weights, both AdamW moments and optimizer
step into PyTorch before each diagnostic comparison. Packed Q/K/V ownership and scalar parameters are
restored explicitly and checked by byte-exact round trips through the effective-weight mapping.
No oracle value is copied into CKE. The independent trajectory remains a separate required experiment.
For the same small complete mixed head, only zero-based update 18 fails unchanged numerical gates. Maximum gradient, master-weight and moment errors are 6.748e-4, 2.578e-5 and 6.748e-5. This is still numerical FAIL, not complete mixed-precision certification. The earlier independent update-8 failure disappears when master weights and optimizer state are synchronized.
The remaining first significant divergence is hidden_norm → memory_projection.
With identical inputs and normalization parameters, native and PyTorch LayerNorm differ by at most
2.384e-7. One value is 1.0976563692092896 versus 1.09765625; these round to BF16 values 1.1015625
versus 1.09375. Replaying the projection on identical operands gives zero output error, dW error
1.863e-9 and isolated dX error 4.657e-10. The packet retains the LayerNorm operands and the rounding
witness. This exposes amplification at a precision boundary; it does not justify changing global tolerances.
Input gradients may contain contributions from multiple branches. The diagnostic now replays the selected provider's dX into its own buffers rather than comparing it with the accumulated input gradient. A regression deliberately omits the accumulated dX to enforce this distinction.
CKE's forty native losses, final master-weight hash and final optimizer-state hash are identical in same-state and independent runs. Checkpoint/restart remains exact. The report labels same-state comparison explicitly and marks independent trajectory parity NOT_TESTED in that report; the separate independent run remains FAIL. Production kernels, maps and compiler behavior are unchanged by this diagnostic.
python tests/run_clef_training_oracle.py --mixed --same-state \
--report /tmp/clef-oracle/same-state.json
# Keep the independent run alongside it; both currently exit 1.
python tests/run_clef_training_oracle.py --mixed \
--report /tmp/clef-oracle/independent.json
Evidence: docs/notes/clef_same_state_oracle_20261113.json. Next: evaluate the declared
LayerNorm-to-BF16 precision boundary and a justified mixed-precision acceptance policy, preserving the
independent FP32 and mixed trajectory checks. Do not force backend rounding history merely to obtain PASS.
Complete-head PyTorch comparison: FP32 passes; mixed trajectory fails
The complete small topology now runs forty independent native/PyTorch AdamW updates with RNE BF16 projection operands and FP32 masters, bias, gradients, optimizer state and remaining operations. Biased projections use explicit generic forward/backward maps. Backward synthesis still owns the derivative graph; there is no Clef-specific compiler dispatch or Python update of CKE weights.
This is diagnostic mixed-precision training, not mixed-precision certification. The pinned reference owns the head topology; a test-only PyTorch autograd adapter declares the projection precision and identity-rounding VJP. This is not default autocast. Inputs are identical synthetic frozen backbone features, with two questions and unequal option counts. It is not complete backbone training.
| Comparison | Measured result |
|---|---|
| FP32 complete small head, 40 updates | PASS; loss 0.28329045 to 0.06633431; maximum gradient error 1.375e-6. |
| Released FP32 head, 4 updates | PASS; pinned weights, synthetic frozen features; loss 0.59229440 to 0.13316446. |
| Mixed small-head trajectory, 40 updates | FAIL under unchanged bounds; native loss 0.28323126 to 0.06657010. Maximum gradient, weight and moment errors: 0.0328365, 0.00121975 and 0.00366078. |
| Mixed same/fresh-process resume | PASS, exact continuation. Separate generated mixed inference export: NOT_TESTED. |
At zero-based update 8, the first significant projection divergence is
fields0_cross_out. Weight [7,2] is 0.0720214918255806 in CKE and
0.07202146202325821 in PyTorch. Those nearby FP32 values round to different BF16 values:
0.072265625 and 0.07177734375. Replaying the CKE operands in PyTorch yields zero output,
weight-gradient and bias-gradient differences at this boundary; input-gradient error is 1.164e-10.
This identifies a rounding-boundary amplification of prior master-weight drift, not a demonstrated local
projection derivative defect. It does not establish that all later discrepancies are acceptable.
The oracle runner compares raw option logits, loss, every parameter gradient, weights and optimizer moments throughout the trajectory. It retains per-parameter RMS/magnitude metrics, failed gates, artifact hashes and a first-divergence replay packet. Numerical FAIL exits 1; missing assets exit 2. Execution, loss reduction and exact resume cannot override numerical FAIL. Existing bounds are unchanged.
make build/libckernel_engine.so
export CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py
python tests/run_clef_training_oracle.py --report /tmp/clef-oracle/fp32.json
CKE_CLEF_FLASH_HEAD_PATH=/path/to/pinned/joint_head.safetensors \
python tests/run_clef_training_oracle.py --released --report /tmp/clef-oracle/released.json
# Expected exit 1: complete diagnostic comparison, known numerical FAIL.
python tests/run_clef_training_oracle.py --mixed --report /tmp/clef-oracle/mixed.json
python -m unittest tests.test_v8_clef_mixed_training -v
Recorded measurements and artifact identities: docs/notes/clef_full_training_oracle_20261113.json
(Intel Core i7-14700T, CPU PyTorch 2.12.1; no Spark/GPU or performance comparison).
Standard nightly adds biased-provider rejection/autograd, full-topology backward-generation and report-verdict regressions to the existing projection suite. That suite does not certify the failing mixed trajectory. Next: distinguish local same-state update accuracy from independent BF16 trajectories before proposing any mixed-precision acceptance policy. Released mixed training, mixed inference export, LoRA, quantized training, hardware speed and generalization remain unproven.
Bounded BF16 projection training with FP32 masters
One mixed-storage projection graph is certified; complete BF16 Clef training remains unsupported.
The authoring helper version/v8/training/bf16_projection.py replaces an explicitly selected projection
with a map-declared provider. Ordinary backward synthesis creates the VJP and accumulation edges; the existing
generated trainer performs all forward, backward and AdamW arithmetic. There is no model-name compiler branch.
The precision policy is explicit: FP32 inputs and master weights are rounded at each forward projection boundary to
BF16 using round-to-nearest, ties-to-even. Those saved compute operands occupy typed uint16_t buffers.
The provider computes ascending-index FP32 products and sums without FMA contraction and publishes FP32 output.
Backward consumes FP32 output gradients and the saved BF16 operands, producing FP32 input and master-weight
gradients. Operand rounding uses a declared identity derivative; this is not the derivative of a discontinuous
quantizer, nor a claim of equivalence to default PyTorch autocast. The independent reference uses explicit BF16
rounding and FP32 matmul/autograd with the same chosen VJP. It does not call the CKE kernels.
Master weights, gradient accumulation and AdamW moments remain FP32. Frozen projections still propagate activation gradients, but have no parameter-gradient publication or optimizer state. Compute operands refresh on every forward call after updates or checkpoint restore; derived caches are not checkpoint payloads. The runtime admits BF16 only for nonpersistent, nondifferentiable auxiliary tensors with a typed forward producer and typed map consumers. BF16 model weights, BF16 gradient buffers and a global BF16 training configuration still fail explicitly. Layout byte extents and pointer types are checked; canaries reserve whole 32-bit words around packed caches, with direct kernel tests checking the exact 16-bit tail.
| Generated experiment | Evidence |
|---|---|
| 4-channel projection and frozen bridge | 40 native updates: loss 1.42442214 to 0.03998315. |
| 5-to-7 projection, frozen 7-channel bridge | 40 native updates: loss 1.37150788 to 0.00395414; odd BF16 cache extents. |
| Independent trajectory comparisons | Worst absolute gradient error 5.41e-8, master-weight error 9.54e-7, moment error 2.43e-8 (PyTorch 2.12.1 CPU, seed 837). |
| State | Byte-unchanged frozen weights, exact same-process and fresh-process checkpoint continuation, trained-weight forward reload. |
| Primitive contracts | Rectangular/tail shapes, O0/O3 equality, FP32 dX/dW oracle, nonfinite and finite-overflow rejection, null/alignment/overlap/capacity rejection, unchanged published outputs and recovery. |
make build/libckernel_engine.so
python -m unittest tests.test_projection_bf16_train_kernel tests.test_v8_bf16_projection_training -v
python scripts/nightly_runner.py --tests training_projection_bf16 training_selective_parameters --ci --require-executed
Missing PyTorch or the built native engine reports SKIP, not parity PASS. This scalar implementation establishes semantics, not acceleration or an end-to-end memory reduction. It retains FP32 external activations in addition to BF16 saved compute operands. Full-head mixed precision, BF16 normalization/attention policies, public training-job admission, LoRA and quantized-base training remain separate gates. The complete small-head diagnostic above now composes this policy but fails independent trajectory parity; released mixed-head certification remains pending.
Initial BF16 audit and remaining full-head prerequisites
Complete BF16 head training remains unsupported. The initial audit found that the trainer represented all floating tensors as FP32. The bounded projection implementation above adds typed BF16 saved operands while retaining FP32 master hydration, snapshots and updates. The planner recognizing a two-byte BF16 dtype does not establish a typed runtime ABI.
A controlled pre-fix reproduction supplied an FP32 layout and marked a trainable weight BF16.
Codegen emitted a float* buffer, reported no skipped operations, and silently omitted the
weight from parameter_gradient_order and optimizer updates. Storage admission now rejects
unsupported tensor and reference dtypes before C emission. It also rejects inconsistent declarations,
integer weights/gradients, and configurations claiming BF16 with FP32 tensors. The FP32 head author
requires explicit FP32 arrays instead of interpreting packed uint16 weights as numeric FP32 values.
The following table records the initial audit; the projection milestone above resolves its bounded typed-cache case, not every full-head precision boundary.
| Boundary | Existing evidence at audit | BF16 composition gap identified |
|---|---|---|
| Generated training ABI | FP32 buffers and INT32 metadata; FP32 updates and snapshots. | Typed BF16 buffers, byte extents, initialization, saved tensors and checkpoint/export representation. See version/v7/scripts/codegen_train_runtime_v7.py. |
| Gradient synthesis | Existing backward lowering constructs FP32 gradient and temporary tensors. | Explicit gradient precision and conversion edges; no inferred casts from component names. |
| Span mean, exact-erf GELU, grouped pooling | Checked BF16-storage VJPs in src/kernels/head_backward_bf16.c, three maps and independent leaf tests. | The maps lack generated-training contracts. These variants consume and publish BF16 gradients; they cannot silently stand in for FP32-gradient variants. |
| Normalization and projection | BF16 LayerNorm and mixed MLP backward implementations exist in layernorm_kernels_bf16.c and mlp_kernels_bf16.c. | Map/ABI/scratch and cache contracts for the actual head shapes. The fused MLP uses a tanh GELU approximation; Clef's exact-erf GELU cannot be replaced by it. |
| Noncausal attention and decision branches | FP32 generated-head derivatives; BF16 inference attention providers. | Compatible BF16 training forward/backward, indexed/scalar branches and branch-gradient accumulation with explicit rounding points. |
| Optimizer | adamw_update_bf16 exists with BF16 weights/gradients and FP32 moments. | It is not the proposed FP32-master-weight recipe. Declare master/compute weight ownership and refresh boundaries before connecting updates. |
Original implementation sequence (first step now demonstrated above): first prove typed storage and explicit conversion edges on one generic projection. The initial mixed-precision target uses BF16 projection operands, FP32 normalization/softmax/loss/scoring, FP32 master parameters, FP32 parameter-gradient accumulation and FP32 AdamW moments. Each provider retains its declared accumulation order; BF16 storage alone does not imply a particular compute instruction. Specify activation-gradient rounding separately and compare the exact declared operations against PyTorch; do not assume equivalence to arbitrary autocast settings. Then connect normalization, noncausal attention, pooling and scoring through the same graph. Certify the small complete head before repeating released-size tests. Existing BF16-gradient providers remain separately named variants rather than being silently repurposed.
The initial admission-only change fixed unsafe dtype handling and defined the missing boundaries; it does not claim a BF16 speedup, memory reduction, generated BF16 update or complete BF16 head PASS.
python3 -m unittest tests.test_v8_training_storage_contract -v
python3 scripts/nightly_runner.py --tests training_selective_parameters \
training_complete_clef_head decision_head_backward_bf16 --ci --require-executed
The full-head oracle lane requires the pinned source described below. Standalone BF16 derivatives retain their numerical gates. Unsupported BF16 training must remain a clear error until typed execution and composed training evidence replace this restriction.
Released head weights: bounded FP32 training parity
The same canonical head topology now runs training at the released dimensions: 4,096-channel features, width 1,024, 16 attention heads, feedforward width 4,096, two evidence layers and four field layers. The head safetensors and independent reference source are checked against the pinned hashes before use. No kernel, derivative, compiler special case or numerical tolerance was changed.
Four native AdamW updates compare every effective parameter gradient, updated weight and optimizer
moment against pinned PyTorch. On the fixed synthetic-feature request, loss falls from
0.59229439 to 0.13316447. Maximum absolute differences are
1.68e-7 for gradients, 9.54e-7 for weights and 2.36e-8
for moments. This measures arithmetic and state correctness, not downstream decision quality or generalization.
The fixture deliberately keeps backbone features and a small output-embedding table synthetic and frozen. It is released-head training with supplied features, not training with a generated backbone or reproducing the publisher's training data. FP32 AdamW uses the existing declared recipe: learning rate 0.001, epsilon 1e-4, weight decay 0.01, beta values 0.9 and 0.999.
A fresh subprocess loads the generated library, restores the checked checkpoint and executes the remaining native updates. Its loss sequence, final weight hash, optimizer-state hash and step count must exactly match uninterrupted execution. The existing same-process recovery, frozen-weight checks, rejection tests and separately generated inference export remain in place.
make build/libckernel_engine.so
CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py \
CKE_CLEF_FLASH_HEAD_PATH=/path/to/pinned/joint_head.safetensors \
python3 scripts/nightly_runner.py --tests training_complete_clef_head \
training_released_clef_head --ci --require-executed
The released-weight test has its own asset-backed nightly lane and a 600-second limit. Missing assets produce SKIP; configured files with incorrect hashes fail. The small 40-update trajectory remains a separate fast regression. This asset-backed certificate uses substantial memory and is not a latency benchmark.
Still unproven: generated-backbone feature training, a public dataset/job recipe, complete-head BF16, LoRA/QLoRA and real decision-task accuracy. The next precision milestone must certify storage, accumulation, gradients, optimizer updates and export separately.
Complete head topology: FP32 training on frozen supplied features
The bounded experiment now derives the training graph from the canonical complete-head circuit: two evidence-routing layers, four field layers, and every scoring branch. The existing shared v7/v8 backward synthesizer, memory planner, generated C and native AdamW execute 144 forward operations and 494 backward operations. Python authors the graph and runs the independent oracle; it does not update CKE parameters. No Clef-name branch is added to generic lowering.
On the small geometry (six tokens, eight channels, two questions with two/one options), all 162
effective trainable parameter arrays are compared with the hash-pinned reference class using PyTorch
autograd. Forty updates reduce loss from 0.28329045 to 0.06633431.
Maximum absolute differences over the trajectory were 1.38e-6 for gradients, 1.67e-6 for weights,
and 3.15e-7 for optimizer moments (PyTorch 2.12.1 CPU, seed 611).
The test checks gradients, weights, optimizer moments, byte-identical frozen feature/embedding weights,
rejection without updates, exact checkpoint resume, and trained-weight reload through a separately
generated inference entry. Overlapping spans and shared question indices exercise gradient accumulation.
The declared recipe uses FP32 AdamW, learning rate 0.001, weight decay 0.01, beta values 0.9/0.999,
and epsilon 1e-4 in both implementations. This explicit epsilon limits amplification of
roundoff-sized gradients on shift-invariant biases; it does not change the forward or gradient tolerance.
The existing component recipes retain their own epsilon and remain separate regressions.
Reusable additions are checked indexed gather/scatter, explicit per-operation row counts for linear and add derivatives, generated integer metadata initialization checks, and opt-in contiguous flattening. The checked projection source is compiled with the generated library, using its existing scalar/AVX provider contract rather than changing global engine compiler flags.
make build/libckernel_engine.so
CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/pinned/joint_schema_model.py \
python3 -m unittest tests.test_v8_clef_full_training \
tests.test_v8_indexed_rows_training -v
Standard nightly registers training_complete_clef_head. Missing reference source or
PyTorch is explicitly SKIP; structural and native gather/scatter checks do not stand in for oracle execution.
Remaining gates: real generated-backbone
features, a public dataset/job recipe, and complete-head BF16/LoRA/QLoRA
training remain NOT_TESTED here. This fixed-feature low-level composition is not an unrestricted
cke.v8.compile training API or an application-ready trainer.
Real-feature capture preparation — backbone run still NOT_TESTED
The next gate is a released-backbone feature capture followed by the packaged generated head. The local P3 cache currently contains the released head and reference source, but the post-trained Clef 9B backbone was not found. The new tooling does not download a model or substitute ordinary Qwen weights. Actual real-backbone capture and its generated-head numerical comparison remain NOT_TESTED until the pinned assets are supplied.
capture_clef_features.py is development-only PyTorch oracle tooling. It loads a local
release after checking an administrator-pinned file-hash lock, committed head/config/source identities,
and every indexed backbone shard. Tokenizer and other discoverable loader files must also appear in
the lock. The lock must come from independently verified release assets; naming a revision in an
untrusted manifest does not prove weight provenance. No production worker imports this script.
The pinned schema encoder silently truncates state at its length limit. This wrapper first encodes without truncation, rejects requests exceeding the declared capacity, then verifies the bounded result is identical. Images/video are explicitly unsupported. It retains original token IDs, question/option spans, stable candidate IDs and request hash. The backbone runs in BF16 with eager attention; actual hidden states and output-embedding rows are cast to FP32 for the same-input head comparison. This is not a claim of BF16 head parity or full-model FP32 parity.
Only used embedding rows are retained, with their original token IDs. The head-local IDs are remapped without changing their span positions or values. A nonzero complete-head PyTorch test proves that this compaction preserves every compared intermediate and final logit exactly. Capture loading checks byte hashes, token reconstruction, question ownership, expected reference shapes and finite reference values before native replay. Private requests/features remain local artifacts, not routine logs.
replay_clef_features.py builds a package using the existing authoring and compiler path,
then compares four field-layer outputs, final normalized fields and all raw option logits. Intermediate
absolute tolerance stays 1e-5; raw logits stay 2e-6, relative zero. It also compares each returned question's
scores, retaining IDs and unequal option counts. Question zero → other questions → question zero tests
selection and reuse within one captured context; it is not a substitute for distinct real-input A→B→A
coverage. A package still has one compiled geometry. The existing 1,000 ms worker deadline is preserved;
a deadline rejection is reported as execution failure with numerical NOT_TESTED, not silently bypassed.
# release-lock.json: {"model":"Cloudflare/clef-flash", "revision":"PINNED_REVISION",
# "files":{"config.json":"SHA256", ...every required local file...}}
# Obtain the lock and its expected SHA independently from verified release assets.
python3 -m version.v8.decision.capture_clef_features \
--release /trusted/clef-release --release-lock /trusted/release-lock.json \
--release-lock-sha256 LOCK_SHA256 --request request.json \
--output /private/clef-capture --max-tokens 512 --threads 4
python3 -m version.v8.decision.replay_clef_features \
--capture /private/clef-capture --capture-sha256 CAPTURE_JSON_SHA256 \
--head /trusted/clef-release/joint_head.safetensors --output /private/clef-replay
# Cheap contracts; reference-source test skips explicitly when absent.
python3 -m unittest tests.test_v8_clef_feature_capture -v
# Real backbone gate is optional and must report SKIP without these assets:
CKE_CLEF_RELEASE_PATH=/trusted/clef-release \
CKE_CLEF_RELEASE_LOCK=/trusted/release-lock.json \
CKE_CLEF_RELEASE_LOCK_SHA256=LOCK_SHA256 \
python3 -m unittest tests.test_v8_clef_feature_capture.RealBackboneCaptureTest -v
The standard decision_feature_capture nightly lane registers both the cheap contracts
and the explicitly asset-dependent real-backbone gate. Five local contract tests passed, including
pinned-source truncation detection; the real-backbone test was skipped. This prepares evidence collection;
it does not advance the CKE backbone, head training, LoRA or end-to-end decision certification.
Registered generated head: experimental loopback evaluation
The cke.decision.evaluate.v1 proposal now connects one registered generated
head package to the independent simulated Antsand consumer over real HTTP. This is
head_only / native_generated, not text-to-decisions. The existing synthetic fixture
lane remains explicitly contract_simulation / fixture_replay. Neither executes
business actions. The read-only cke.control.v1 protocol is unchanged; no real Antsand
service or running deployment is modified.
| Boundary | Status |
|---|---|
| Supplied features → generated complete head → raw option logits | Implemented through a persistent loaded package and one generated native entry. |
| Released head weights over HTTP versus pinned reference | PASS on P3; synthetic backbone features and embedding rows, two distinct inputs and A→B→A. |
| Text preparation / actual generated backbone | NOT_IMPLEMENTED in this evaluation path. |
| Compute cancellation, remote authorization, real Antsand integration | NOT_IMPLEMENTED. Consumer cancellation suppresses publication; it cannot preempt C. |
| Head training | Existing derivative/optimizer work is unchanged; this is forward inference integration. |
The offline builder version/v8/decision/package_clef_head.py uses the ordinary
circuit → map resolution → lowering → generated-C compiler. It packs effective FP32 weights and
registered inputs; it performs no model arithmetic. Shared checked-graph build code now lives in
version/v8/scripts/compile_checked_bundle_v8.py, with compatibility imports for existing
tests. The worker uses only Python's standard library for request handling and byte copies; all
head arithmetic remains in the generated entry. A subprocess test uses python -S to
prove NumPy, safetensors and PyTorch are not runtime dependencies.
An administrator supplies a trusted package manifest and its expected SHA-256. This is executable native code, not an untrusted upload interface. The manifest binds the model revision, CKE revision, effective weights, circuit, layout, call IR, generated source, compiler/build identity, native library and registered feature files. The loader checks hashes, shapes, offsets, disjoint input/output/weight regions, finite features, token IDs, spans, final-valid-token selection, option counts and candidate ownership before loading C. On Linux, it loads the exact verified library bytes from a sealed memfd. Weights load once; each request clears non-weight arena regions and copies all of its registered inputs. The host does not schedule neural operations. Packages have one compiled geometry and at most 64 registered artifacts; runtime-variable feature shapes and arbitrary uploads are not supported.
Each registration contains hidden, input_ids,
question_spans, option_spans, global_span,
option_counts and question_type_ids, plus a selected question index and its
stable candidate IDs. The entire multi-question context executes; one question's logits are returned.
Reordering request candidate IDs only reorders those scores. Changing the head context requires a
new registered artifact and hash. artifact_id in response evidence is the package SHA-256;
ir_id is the call-IR SHA-256; library_sha256 identifies the loaded bytes.
The input hash binds both feature bytes and the registration metadata. No host paths are returned.
The loopback worker allows one active evaluation, no evaluation queue and four connection threads. Concurrent evaluations return BUSY; excess connections close. A two-second absolute connection timer prevents slow clients from occupying threads indefinitely. Requests and responses remain bounded to 16 KiB, 16 candidate IDs and a maximum 1,000 ms consumer attempt. The worker checks expiration before and after native execution and before responding. The independent consumer rechecks the same absolute monotonic deadline inside its publication lock, after validation and lock wait. Cancellation/replacement invalidates pending results immediately. A disconnected or expired native call retains its slot and arena until C actually returns. Closing the worker joins handlers before unloading its library. Remote exposure, authentication and native interruption require separate contracts.
Reproduction
Prepare a local registrations JSON list with id, npz,
question_index and candidate_ids. Each NPZ contains exactly the seven inputs
above, hidden FP32 and other arrays int32, matching the declared geometry. Supply pinned released head
weights and the intended output-embedding rows as an NPY. Record whether features and embeddings are
synthetic or actual backbone captures; a pinned head alone does not establish either.
python3 -m version.v8.decision.package_clef_head \
--head joint_head.safetensors --head-sha256 HEAD_SHA256 \
--embedding embedding.npy --embedding-sha256 EMBEDDING_SHA256 \
--registrations registrations.json --revision MODEL_REVISION \
--output /trusted/head-package
python3 -S -m server.decision.native_server \
--package /trusted/head-package/package.json --package-sha256 PACKAGE_SHA256
# Copy one printed bounded request object to request.json; use the printed loopback URL.
python3 -m server.decision.consumer --url http://127.0.0.1:PORT/evaluate --request request.json
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
server.tests.test_decision_native server.tests.test_decision_evaluate -v
python3 scripts/nightly_runner.py --tests decision_native_head decision_evaluate_contract --ci
# Optional independent released-head oracle lane:
CKE_CLEF_FLASH_HEAD_PATH=/path/joint_head.safetensors \
CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/joint_schema_model.py \
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
server.tests.test_decision_native.PinnedNativeDecisionTest -v
The server prints fresh boot IDs and example requests; a restart invalidates old IDs. Do not treat this local descriptor as an approved remote discovery/authentication protocol. No persistent service is installed. Registered features may be private; retain packages accordingly.
Evidence and remaining gates
The asset-free nonzero synthetic head test compares returned logits to an independent PyTorch
composition at absolute tolerance 2e-6, relative zero. It covers A→B→A versus a fresh runtime,
candidate permutation, frozen weight bytes, sealed library bytes, invalid package identity/layout,
native failure/recovery, BUSY, deadline expiration and cancellation during actual native execution.
The existing fixture responder continues testing malformed protocol, restart and publication-lock
behavior. Missing PyTorch or pinned assets are explicit SKIPs. Add --require-executed to nightly only with the required assets configured; it correctly fails on a missing-asset subtest even when the remaining tests pass.
On P3, released head/source identities from the committed Clef fixture were verified. Two distinct 6×4096 synthetic feature arrays, synthetic 16×4096 output embeddings, two questions and three options ran A→B→A through the packaged worker. Returned question-zero logits had maximum absolute error 2.39e-7 against the pinned direct reference (unchanged acceptance 2e-6); repeated A was exact. Native request execution was 234–265 ms in this run, excluding load/build. This is not a performance benchmark or end-to-end backbone result. Runtime responses always retain numerical NOT_TESTED: passing an independently run oracle test does not certify every subsequent request.
Next: agree on the execution contract with Antsand, register captured real backbone features and embeddings, then connect generated backbone execution. Keep head-only, full-model, transport, training and authorization verdicts separate. No decision score grants permission to act.
Indexed scoring gradients: separate normalization contracts
The final scoring branches now have checked FP32 derivatives for indexed pair features [a, b, a·b, |a−b|], normalized-vector dot products and cosine similarity. Owner indices route each option to its question. Repeated owners accumulate contributions; unreferenced owners receive zero. Both public gradient buffers are published only after all checks and reductions succeed. Caller-owned scratch holds groups·channels + 2·rows·channels floats; there is no kernel allocation.
Forward equivalence does not establish backward equivalence. The pinned head uses normalize-and-dot for its prior score and cosine_similarity for its joint score. PyTorch 2.12.1 normalization has a zero clamp derivative below epsilon, whereas its cosine implementation clamps a cloned norm under NoGrad and retains the norm derivative below epsilon. The new providers encode those policies separately. At zero norm, the norm VJP is zero; the direct numerator derivative still flows. At the normalization clamp boundary the derivative is one. Absolute difference uses zero subgradient at equality, including signed zero.
Consequently the cosine provider's below-clamp VJP deliberately matches the pinned framework's backward convention, not the finite-difference derivative of its clamped forward function. Central differences are tested away from nonsmooth/clamped boundaries; separate autograd tests cover zero, below epsilon, equality and adjacent FP32 values on both operands. Future head backward authoring must select the explicit derivative contract. Neither forward provider identity nor epsilon alone selects it.
| Independent PyTorch 2.12.1 CPU comparison | Largest observed absolute gradient error |
|---|---|
| Indexed pair features | 2.39e-7 |
| Normalized-vector dot | 2.24e-8 |
| Cosine similarity | 2.24e-8 |
Measured 2026-11-08 on synthetic nonzero operands, widths 1, 7 and 129, with repeated and unreferenced owners. Primitive PyTorch ceilings are absolute 2e-6 / relative 2e-5; generated joined gradients use 3e-6 / 3e-5. These are empirical fixture bounds. Ordered scalar tests also cover width 64, O0/O3 equality and the reduction policy: each row contribution rounds FP32, then repeated-owner contributions sum in ascending row order in FP64 before the final FP32 store. An adversarial [1e8, 1, −1e8] scatter retains one; this does not claim bitwise equivalence to FP32 index-add.
One ordinary generated backward entry executes pair-feature/GELU and both cosine branches, then joins their left/right gradients using the existing checked addition provider. It compares the combined VJP against PyTorch, exercises A→B→A arena reuse, rejects insufficient arena capacity and a late nonfinite branch, and recovers. A deliberately miswired branch join produces a detectable gradient error. Primitive tests cover null/misaligned pointers, every short capacity, partial writable aliases, invalid owners, zero/overflow dimensions, invalid epsilon and late finite scatter overflow.
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
tests.test_v8_decision_scoring_backward -v
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
--tests decision_scoring_backward --ci --require-executed
This is generated scoring-component backward, not full-head training. The circuit explicitly authors the derivative branches; automatic complete-head backward, scalar/gain and remaining indexed derivatives, grouped losses, all-parameter AdamW trajectory, BF16 optimizer certification and adapters remain separate gates. Existing full-head forward and attention-training regressions remain independent evidence. No Clef-specific compiler branch was added.
Noncausal attention backward and native training
A bounded generic composition now trains through feature lookup → trainable projection → Q/K/V projections → full noncausal attention → residual → scores → cross-entropy. Automatic backward lowering declares every branch accumulation and native AdamW updates the selected parameters. The new provider handles unequal query/memory lengths, packed token-major heads and caller-owned scratch. It does not implement masking, dropout or BF16 attention backward.
Forward arithmetic remains FP64 ascending dot products with FP32 score/exponential stores. Backward recomputes those scores; dS stores FP32, dQ reduces FP64, and shared dK/dV accumulate in ascending query order with an FP32 store after each query. All three gradient outputs are staged until validation and arithmetic succeed. Scratch is Q·H·D + 2·S·H·D + 2·S floats.
The shared trainer now propagates map-declared zero-success status. A checked kernel failure stops the step before AdamW, leaves weights/moments untouched, clears the incomplete accumulation window and leaves the caller's loss output untouched. This is not general rollback for arbitrary void providers. The fixed-batch contract rejects partial batches rather than allowing padding into unmasked softmax; native kernel tests cover unequal Q/S, while the composed training fixture uses four tokens. Variable-length masked training remains a separate contract.
| FP32 native AdamW, 40 updates | Initial → final loss | Gradient / weight / moment max absolute error |
|---|---|---|
| All Q/K/V trainable | 1.32838511 → 0.00033159 | 1.75e-7 / 9.54e-7 / 5.22e-8 |
| Frozen K projection | 1.32838511 → 0.00041567 | 1.20e-7 / 9.54e-7 / 3.73e-8 |
Measured 2026-11-07 with PyTorch 2.12.1+cpu and seed 837. The oracle uses explicit independent softmax attention and autograd, not Unsloth. Ordered scalar FP64/FP32 comparisons at O0/O3 and native forward finite differences provide additional checks. Gradient/weight ceilings remain 3e-5 absolute, 3e-4 relative; optimizer moments 3e-6 absolute, 3e-4 relative. These are bounded empirical checks, not a universal error bound or bitwise PyTorch claim. Reports retain engine, generated-C and runtime-contract hashes.
Checkpoint reset/resume and trained-weight inference reload pass. Frozen K weights remain byte-identical while gradients reach the upstream trainable projection. Swapping the saved K input with V fails the gradient comparison. An injected late backward failure after one successful microstep clears partial gradients; two subsequent microsteps exactly match a clean two-microstep baseline. Null/invalid IDs, labels, batch lengths, undersized capacities, nonfinite operands, aliases, misalignment and finite accumulation overflow have rejection coverage.
make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
tests.test_v8_attention_cross_backward tests.test_v8_attention_training -v
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
--tests training_attention_composition --ci --require-executed
Complete Clef-head training remains NOT_TESTED. This certifies a synthetic attention/projection/residual training composition. Next: connect existing head VJPs and the remaining indexed/scoring derivatives and grouped loss, then certify every released-head gradient and update. BF16 optimizer trajectories, LoRA and quantized-base training are separate milestones.
Cached-statistics LayerNorm training composition
The next trainable composition is fixed features → trainable projection → affine LayerNorm → trainable scores → cross-entropy. Existing native LayerNorm forward/backward arithmetic is reused; no new normalization kernel or model-specific compiler branch is introduced. The forward provider publishes its actual FP32 mean and reciprocal standard deviation, which automatic backward lowering passes to the existing VJP. Epsilon belongs to the operation, independently of the enclosing model.
Shared training code generation now accepts an opt-in, map-owned ABI for bounded contiguous FP32 buffers and literal scalar parameters. It rejects missing ports/providers, reordered ABIs, unsupported phases/dtypes, invalid scalars, insufficient capacities, incorrect shapes and writable aliases instead of guessing argument bindings. The selected maps participate in runtime/checkpoint identity. The legacy LayerNorm providers remain void: finite operands are their caller precondition. Checked providers now use the separate zero-success failure contract described above.
| FP32 native AdamW, 40 updates | Initial → final loss | Gradient / weight / moment maximum absolute error |
|---|---|---|
| Trainable affine, epsilon 1e-5 | 1.29325175 → 0.00443579 | 1.98e-7 / 9.54e-7 / 5.59e-8 |
| Frozen affine, epsilon 0.03 | 1.31046605 → 0.05293253 | 8.95e-8 / 6.26e-7 / 3.36e-8 |
Measured on 2026-11-07 against PyTorch 2.12.1 CPU, seed 837. Every trainable gradient, weight and AdamW moment is compared at every update. Frozen weights remain byte-identical; an upstream finite difference checks gradient flow. Resetting and restoring a step-20 v8 checkpoint reproduces remaining updates exactly, and final-weight reload reproduces forward-only logits. A deliberately swapped mean/rstd capture fails the gradient oracle. Direct native VJPs also cover widths 1, 7, 512, 513 and 4096. These are synthetic operands, not released Clef-head training or backbone features.
Numerical ceilings retain absolute 3e-5 and relative 3e-4 for gradients, weights and logits; moments use absolute 3e-6 and relative 3e-4. They are empirical fixture bounds. The existing forward uses compile-target-dependent FP32 SIMD variance partials and FP64 merging; backward reduces in FP32. Neither cross-compiler bitwise agreement nor BF16 optimizer certification is claimed.
make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest \
tests.test_v8_layernorm_training tests.test_v8_selective_training -v
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
--tests training_layernorm_composition training_selective_parameters --ci --require-executed
This change also aligns the shared v7 attention-score derivative scratch shape with the saved attention tensor, matching the v8 rule. It resolves the inherited attention-rematerialization shape assertion reported in the earlier selective-training sweep.
This does not yet make the complete Clef head trainable through cke.v8.compile. It extends the existing backend, maps and v8 derivative rules. Remaining work includes head authoring into the training IR, integration of the certified noncausal attention VJP, indexed/scoring derivatives, grouped losses and complete head parameter/optimizer/export comparisons. BF16 head trajectories and LoRA/QLoRA follow that baseline.
Automatic pooling derivatives and typed training inputs
Span means and grouped softmax-weighted pooling now use declared FP32 training contracts and v8 derivative rules. The shared lowerer generates their VJPs and explicitly accumulates gradients from separate branches. Integer spans and group counts are nondifferentiable; overlapping spans sum their feature-gradient contributions. Existing kernel arithmetic and inference ABIs are unchanged.
Fixed-batch training graphs declare config.training_integer_inputs, an ordered list of
INT32 input tensor IDs. Generated summaries expose integer_inputs (tensor, offset, count,
dtype) and integer_input_count. These bindings participate in runtime/checkpoint identity.
The host packs metadata in that order and calls:
ck_train_step_with_i32_ex(token_ids, targets, valid_tokens,
integer_inputs, integer_count, loss_out, learning_rate)
The generated runtime copies metadata for the current batch and owns the forward/backward/optimizer
schedule. The caller retains ownership of its source buffers. The old step entry cannot silently reuse
metadata from a previous batch: it rejects graphs requiring these inputs. Forward-only inspection uses
ck_train_set_batch_with_i32_ex followed by the existing generated forward entry.
Counts, alignment and exact input extents are checked before binding. Providers validate span ranges and positive group counts with exact option coverage. Compile-time checks reject missing metadata, trainable integer inputs, incompatible shapes/dtypes, wrong contiguous strides, and partial-token span training. This initial contract requires fixed capacities with all token rows valid; it does not certify arbitrary dynamic shapes. The existing generated runtime remains non-reentrant and must be serialized per loaded instance.
The regression composes fixed features → trainable projection → overlapping spans → exact-erf GELU → grouped weighted pooling → frozen projection → scores. The projection also feeds the grouped queries, so backward must add both branches. Six spans form unequal groups [2,1,2,1]. Forty native AdamW updates reduce loss from 1.40194964 to 0.16358216. Against PyTorch 2.12.1 CPU, maximum absolute differences are 8.01e-8 for gradients, 4.77e-7 for weights and 3.36e-8 for moments. Existing fixture bounds are unchanged.
Tests compare every trainable gradient, update and moment, upstream finite differences, frozen weights, same-process checkpoint restoration and inference reload. A wrong saved-query control fails the oracle. A→B→A inputs, caller-buffer mutation after binding, malformed metadata, nonfinite input and injected backward failure cover reuse and recovery. Admission rejection leaves the current accumulation intact; provider failure discards its partial accumulation window and recovery matches fresh execution. PyTorch absence is an explicit oracle skip.
make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_pooling_training -v
python3 scripts/nightly_runner.py --tests training_pooling_composition --ci --require-executed
This does not register complete Clef training. Indexed/scalar scoring derivatives, grouped decision loss, full-head training authoring and released-head gradient/update/export evidence remain. BF16 training, LoRA/QLoRA and application submission remain separate acceptance gates. The existing job service continues to reject unsupported recipes.
Automatic training through learned gains and combined scoring branches
The FP32 scoring composition now joins normalized-dot prior scores, cosine joint scores and a residual MLP, with two learned exponential scales and a sigmoid gate, before grouped decision loss. Its derivatives are selected by v8 rules and map contracts; the existing shared lowerer emits branch accumulation, parameter gradients and native AdamW. This milestone adds no generic compiler logic or model-specific backward scheduler.
Three reusable checked VJPs cover scalar exp-clamp, scalar sigmoid and vector scaling. Exp-clamp has derivative one at the clamp boundary and zero above it. Sigmoid differentiates the rounded stable forward value. Vector scaling computes input gradients and an ascending FP64 gain reduction, staging both outputs before publication. Caller-owned scratch, capacities, finite values, alias rejection and failure atomicity are explicit. Existing cosine/normalized-dot VJPs and checked addition gain training bindings. Their distinct below-epsilon norm derivatives remain explicitly selected; the forward result alone cannot identify which derivative is appropriate.
The nonzero model-neutral fixture uses two questions with one and three options, repeated owner indices and overlapping spans. All three learned scalars have significant initial gradients and independent finite-difference checks. Every trainable parameter gradient, weight and moment is compared with PyTorch through 40 native AdamW updates. The frozen feature table and redundant common score bias remain byte-identical. Checkpoint restore reproduces the remaining trajectory exactly, and trained-weight reload reproduces scores. An injected sigmoid-VJP failure after partial accumulation preserves weights and moments, discards incomplete gradients and recovers to the exact fresh-run update.
Measured on P3 with PyTorch 2.12.1 CPU, seed 837: loss 0.77345836 → 0.00013260. Maximum absolute errors were 2.38e-7 for gradients, 7.16e-7 for weights and 1.09e-7 for moments. Existing regression ceilings remain unchanged. Primitive tests cover independent ordered FP64 arithmetic, O0/O3, PyTorch, finite differences, saturation/clamp boundaries, finite overflow, one-element-short capacities, guards, partial overlap and rejection/reuse. The explicit FP64 gain test sums [1e8, 1, -1e8] to one.
The unchanged complete-head forward test also passes with hash-verified released head weights and pinned source at six and eight synthetic feature tokens: maximum raw-logit errors are 1.19e-7 and 1.79e-7. This protects forward inference; it does not certify released-head backward execution.
make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_scoring_training -v
python3 scripts/nightly_runner.py --tests training_scoring_composition --ci --require-executed
Complete released-head training remains NOT_TESTED. This is a reduced scoring graph, not the four field layers and evidence stack trained together. Next assemble those existing training components into complete head authoring, audit remaining broadcast/selection and trainable-embedding bindings, and certify every head gradient, update and export with released weights. No new application recipe is exposed. BF16 composed training, backbone execution, LoRA and QLoRA remain separate gates. The current trainer requires fixed capacities and positive scalar constants; the standalone exp-clamp kernel's finite-maximum domain is wider. Missing PyTorch or a native engine is reported as SKIP.
Declared grouped decision loss and indexed scoring training
A training graph can now declare its scalar loss and score tensor explicitly instead of requiring
an operation named logits. The selected map declares the gradient input, gradient output,
scalar value and reduction identity. The existing shared backward lowerer seeds that tensor and applies
the normal derivative rules and branch accumulation. The existing generated native runtime performs
forward, backward and AdamW; no model-name dispatch or separate decision training loop is added.
The new checked FP32 grouped cross-entropy provider accepts positive option counts summing exactly to the score count and one relative label per question. Loss is the mean over questions, including zero-loss singleton questions. It uses ascending FP64 max-shift exponential reductions and FP32 output stores. Loss and gradients publish only after validation. Scratch, typed INT32 labels/counts, fixed capacities and numerical identity are declared; BF16, ignore-index, weighting and double backward are not supported. Existing decoder graphs retain their implicit cross-entropy seed.
The small nonzero regression composes frozen features → trainable projection → overlapping spans and indexed pair features → biased projection → erf GELU → scores → grouped loss. It tests repeated owner indices, unequal option counts, all trainable gradients, an upstream finite difference, 40 native AdamW updates, frozen bytes, optimizer moments, checkpoint restoration and trained-logit reload. Invalid question counts or labels after a partial accumulation window must discard that window without updating weights or moments; subsequent updates must match a clean runtime.
The final scalar score bias is explicitly frozen: adding the same offset to every option cannot change cross-entropy. During bring-up, near-zero FP32 PyTorch reduction residuals caused AdamW to move that redundant bias differently (about 0.02246 on the first update). This fixture does not certify a trainable shared score bias. The hidden projection bias remains trainable. No tolerance was widened.
On P3 with PyTorch 2.12.1 CPU, the bounded fixture loss fell from 0.55899101 to approximately 3e-8. Maximum absolute errors across 40 updates were 1.94e-7 for gradients, 6.56e-7 for weights and 5.96e-8 for moments. The existing absolute ceilings remain 3e-5 for gradients/weights/logits and 3e-6 for moments, with relative 3e-4. Kernel tests also use an explicit FP64 scalar oracle, finite differences, O0/O3 comparison, padding and malformed-buffer rejection. Missing PyTorch or engine assets are skips.
make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_grouped_loss_training -v
python3 scripts/nightly_runner.py --tests training_grouped_loss_composition --ci --require-executed
This is component training, not complete Clef head training. Remaining work includes
complete head training authoring (the combined scoring update above now covers scalar/gain and cosine derivatives),
released-head gradient/update/export comparisons and the application recipe. BF16 composed training,
backbone integration, LoRA and QLoRA remain separate gates. The job service still rejects unsupported
Clef recipes. These low-level declarations do not yet make arbitrary inference circuits trainable through
cke.v8.compile.
Exact-erf GELU in automatic backward lowering
The existing FP32 checked GELU forward and derivative now expose contiguous, fixed-batch training contracts. A v8 derivative rule declares the saved pre-activation, incoming gradient and scratch extent; the shared backward lowerer inserts the VJP and explicit gradient accumulation. There is no handwritten Clef backward scheduler and no new arithmetic kernel. The shared training library now links the existing GELU and head-VJP implementations.
Derivative rules may explicitly name their read ports, preserving existing provider ABIs. Legacy positional names remain supported. Duplicate, missing or invalid names are rejected. Strict non-partial lowering now rejects unresolved backward inputs instead of producing an incomplete training graph. Declared contiguous row strides are checked against tensor widths before C emission.
The model-neutral regression trains fixed features → projection → exact-erf GELU → frozen projection → option scores with the existing native cross-entropy and AdamW. It compares every trainable gradient, weight and moment with independent PyTorch, checks an upstream finite difference, frozen-weight identity, 40 updates, same-process checkpoint restoration and trained-logit reload. A late injected VJP failure must discard partial accumulation and preserve weights/moments; recovery must match a fresh run. This is a small FP32 composition, not released Clef training, BF16 training or a new job recipe.
Measured on P3 with PyTorch 2.12.1 CPU: loss 1.40563822 → 0.18035129 over 40 native updates; maximum absolute differences were 9.32e-8 for gradients, 7.16e-7 for weights and 4.48e-8 for moments. The unchanged fixture ceilings are absolute 3e-5 for gradients/weights/logits and 3e-6 for moments, with relative 3e-4. PyTorch absence explicitly skips the independent comparison.
make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_gelu_training -v
python3 scripts/nightly_runner.py --tests training_gelu_composition --ci --require-executed
Complete Clef training remains unsupported. The complete inference circuit still now has indexed/scalar and grouped-loss component evidence; complete graph authoring remains. Training authoring must then connect that graph to the existing workflow. Only after complete gradients, updates and export/reload pass should the local job registry advertise a Clef head recipe. Backbone-feature capture and frozen-backbone execution remain separate evidence levels.
Selective FP32 training prerequisite
The shared v7/v8 backward lowerer now honors boolean parameter requires_grad
declarations without cutting activation gradients through a frozen operation. Conflicting declarations
on the same shared weight are rejected before emission. Undeclared weights retain legacy trainable
behavior. Operation differentiability remains separate from parameter trainability: a frozen projection
between trainable operations must still participate in backward execution.
Frozen parameters have no canonical parameter gradient, optimizer moment allocation or AdamW update. Existing fused GEMM backward providers still require temporary dW/db output buffers, so these derivatives are computed into discard scratch. This is a correctness contract, not a claim that all unnecessary backward arithmetic or scratch has been eliminated.
A small generic linear decision fixture executes fixed feature lookup → trainable projection → frozen projection → trainable scores → cross-entropy through the existing training IR, memory planner, generated C and native AdamW. This fixture is not the complete Clef head and is not a new public feature-head authoring API. No Python code updates CKE weights.
On 2026-11-07, 40 FP32 native updates reduced loss from 1.42433834 to
0.04014646. Every trainable gradient, updated weight and optimizer moment was compared
with independent PyTorch CPU execution. An upstream finite difference verifies gradient flow through
the frozen middle projection; feature and bridge weights remain byte-identical. The existing v8
checkpoint format restores step 20 into a reset runtime and reproduces the remaining 20 steps exactly.
Reloading final weights reproduces inference logits. Observed maximum absolute errors against PyTorch
were 7.08e-8 for gradients, 7.31e-7 for updated weights and 2.24e-8 for moments.
Regression ceilings are absolute 3e-5 for gradients/weights/logits and 3e-6 for moments, with relative
3e-4. These are bounded fixture contracts, not universal error guarantees. This test does not claim fresh-process resume
or released-head training. PyTorch or native-library absence is an explicit skip.
make -j4 build/libckernel_engine.so
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 tests/test_v8_selective_training.py
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
--tests training_selective_parameters --ci --require-executed
Eight focused tests and the registered nightly lane pass. Both existing decoder training compositions
(2l_dense, 2l_gqa_tail) pass gradients, ten updates, checkpoint/resume
and inference export. A broader 47-test sweep had 46 passes and one attention-rematerialization
shape assertion failure; the same failure was reproduced with the unchanged main lowerer.
Full repository nightly has not been run locally.
Next: connect the remaining complete-head derivatives and grouped loss to this existing training workflow, then certify head gradients, updates and export. BF16 VJP providers are independently tested; this FP32 optimizer trajectory does not certify BF16 head training. LoRA and quantized frozen-base training remain separate, later gates.
Complete bounded head forward pass
build_clef_flash_complete_head_circuit.py executes 140 operations in one generated entry:
head ingress → two evidence-routing layers → option summaries → field assembly → four pre-norm
self/cross-attention decoder layers → field/option normalization → prior, cosine and residual scoring
→ raw option logits. The host supplies hidden features, output-embedding rows, token IDs, checked spans,
option counts and question types. It does not schedule the layers.
Question types index the released three-row type embedding. Consecutive option groups retain their question/option order; tests reorder entire questions with their option blocks and type identities. No softmax decision policy is imposed on the final raw scores. There is no generated-backbone or complete service certification in this milestone.
Six reusable checked operations close the scoring gaps: row broadcast addition, indexed pair features (left, right, product, absolute difference), indexed cosine reduction, clamped scalar exponential, stable scalar sigmoid and scalar vector multiplication. Their maps declare sizes, scratch, numerical semantics and status propagation. Generic compiler changes are operation catalogue entries only.
| Pinned head, synthetic inputs | Worst decoder-block absolute error | Final logit maximum / RMSE |
|---|---|---|
| 6 tokens, counts [2,1] | 3.81470e-6 | 1.19209e-7 / 9.73340e-8 |
| 8 tokens, counts [1,2] | 5.72205e-6 | 1.78814e-7 / 1.06067e-7 |
Measured 2026-11-07 with PyTorch 2.12.1 CPU, the hash-verified released head and its direct forward implementation. Both sides receive identical synthetic hidden features and synthetic output embeddings. These are two separately compiled bounded geometries, not a dynamic serving interface or captured backbone trajectories. Intermediate absolute regression ceilings are 1e-5; final logits use 2e-6. These empirical bounds do not establish training or other precision contracts.
python3 version/v8/decision/build_clef_flash_complete_head_circuit.py --check
python3 -m unittest tests.test_v8_decision_tensor_ops tests.test_v8_clef_flash_complete_head -v
export CKE_CLEF_FLASH_HEAD_PATH=/path/to/joint_head.safetensors
export CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/joint_schema_model.py
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 scripts/nightly_runner.py \
--tests decision_tensor_ops decision_clef_flash_complete_head --ci --require-executed
The independent comparisons require PyTorch; pinned comparisons additionally require safetensors, the reference source dependencies and the two hash-checked assets. Missing dependencies/assets are explicit skips. Synthetic tests cover intermediate logits, two lengths, unequal groups, question permutation, malformed IDs/spans/counts/type IDs, nonfinite hidden input, undersized arena, a deliberately corrupted scoring weight, and A→B→A recovery. A model-neutral six-operation generated circuit exercises all new providers independently of Clef topology.
First generated backward composition
Three reusable FP32 vector-Jacobian products now cover token-span means, exact erf GELU, and grouped query-conditioned weighted pooling. They are backward providers, not inference aliases. Their maps own strides, capacities, scratch, numerical contracts and status propagation. The checked-call emitter now accepts an explicit backward phase and rejects providers used in the wrong phase. No model-name dispatch or second training workflow is introduced.
The model-neutral regression executes one generated forward entry for span pooling → erf GELU →
grouped pooling, then one generated backward entry for the reverse chain. Only native-produced forward
intermediates enter that backward entry. This is an explicitly authored VJP circuit, not automatic
backward synthesis through cke.v8.compile. It does not contain trainable parameters,
a loss, optimizer updates, or the remaining complete-head derivatives.
- Span gradients sum all overlapping contributions in ascending span order with FP64 accumulation. Unreferenced valid tokens become zero; inactive token rows and padding remain unchanged.
- Erf GELU uses its analytic derivative in FP64, distinct from the existing tanh-approximation derivative.
- Grouped pooling differentiates both value and score paths, recomputing the forward provider's FP32 probabilities. Ordered FP64 reductions and explicit FP32 intermediate stores are declared in its map. The options and query gradients publish together only after all checks succeed.
- Each VJP overwrites its valid output gradient. Overlapping contributions inside a VJP accumulate; fan-out across separate graph operations still requires an explicit gradient-accumulation operation.
- All outputs and scratch must be disjoint from read buffers and one another. Scratch may change on rejection; that kernel's gradient outputs do not. A later operation failure can leave earlier graph intermediates written, so the entry status must be checked. Tests inject a final-operation failure and verify that the final token gradient remains untouched, then recover.
Tests cover independent ordered arithmetic oracles, optional PyTorch autograd, finite differences for both grouped-pooling inputs, unequal and singleton groups, padded strides, null/misaligned/overlapping regions, capacity and finite-overflow rejection, and A→B→A recovery. Generated execution and scalar oracles require no model assets or PyTorch. PyTorch absence is an explicit SKIP for the autograd comparison.
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_head_backward -v
python3 scripts/nightly_runner.py --tests decision_head_backward --ci --require-executed
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 version/v8/scripts/run_training_composition_v8.py \
--run-root version/v8/.cache/clef-backward-decoder-regression
| FP32 gradient versus PyTorch 2.12.1 CPU (2026-11-07) | Maximum absolute error | RMSE |
|---|---|---|
| Overlapping-span input | 2.98023e-8 | 6.23360e-9 |
| Erf GELU input | 5.96046e-8 | 3.12435e-8 |
| Grouped pooling options (both paths) | 4.47035e-8 | 1.70069e-8 |
| Grouped pooling query | 1.02445e-8 | 4.22744e-9 |
These are measured small, nonzero synthetic cases, not universal error bounds or released-head gradient
certification. Regression comparisons use absolute 4e-7 / relative 3e-6 against PyTorch, and exact checks
for the explicitly ordered span/grouped arithmetic oracle. The focused 9-test suite passes; the four
selected standard-nightly suites pass with no skips, including the hash-checked released-head forward
comparison. Both existing decoder compositions (2l_dense, 2l_gqa_tail)
also pass 10 native AdamW updates, trajectory/gradient/weight/moment comparisons, fresh-process resume
and inference export. Their aggregate report was /tmp/clef-backward-decoder-regression.json.
The full repository nightly is not a local result.
Complete Clef head training remains NOT_TESTED. Next integrate checked biased-linear, LayerNorm and noncausal attention derivatives, indexed/scoring derivatives and branch accumulation into the existing v8 backward workflow. Then certify all head gradients, grouped loss, AdamW updates, frozen-backbone identity, checkpoint/resume and generated inference reload. Existing decoder training is a shared-consumer regression, not evidence that this head can already train.
BF16-storage backward providers
The same three VJPs now also have explicit BF16-storage maps. Saved floating-point operands, incoming gradients and outgoing gradients use BF16; spans/counts remain INT32. Caller-owned FP32 workspace expands BF16 operands exactly, invokes the existing FP32 VJP arithmetic, then rounds each output gradient to BF16 using round-to-nearest-even. The FP64 reductions and FP32 intermediate stores inside the FP32 providers are preserved. There is no intermediate BF16 rounding inside a VJP.
This is a particular mixed-precision derivative contract, not a claim of parity with an arbitrary BF16/autocast forward graph. Grouped probabilities are recomputed using the existing FP32 forward semantics on expanded BF16 saved operands. Independent autograd runs FP32 computation on identical dequantized operands and rounds its gradients to BF16. The tested gradients match that oracle exactly after rounding; this is not a universal bit-identity guarantee.
| Provider | Required FP32 workspace (floats) |
|---|---|
| Span mean | (span_count + token_capacity + valid_tokens) × channels |
| Erf GELU | 4 × rows × channels |
| Grouped weighted pooling | 3 × option_rows × channels + 6 × groups × channels + 2 × option_rows |
Element counts and row strides use the pointed-to dtype, not bytes. BF16 buffers require two-byte alignment; INT32 spans/counts and FP32 scratch require four-byte alignment. Outputs and scratch must be disjoint from all reads and each other. Input padding is ignored, output padding/inactive rows stay untouched, and rejected calls publish no outputs. In particular, a finite FP32 gradient that rounds to BF16 infinity rejects the call before either grouped gradient is published.
The model-neutral generated backward entry composes all three BF16 providers through ordinary maps, lowering and memory planning. Tests include odd channel counts, unequal/singleton groups, ordered oracles at O0/O3, exact/one-short capacities, two-byte-aligned buffers, mixed-width partial overlaps, nonfinite input, FP32 and BF16-only overflow, RNE midpoint ties, subnormal values, final-operation failure propagation, and A→B→A recovery. Saved inputs in this test are synthetic; this does not execute a complete BF16 Clef forward/backward pair. PyTorch absence skips only the independent autograd test.
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m unittest tests.test_v8_head_backward_bf16 tests.test_v8_head_backward -v
python3 scripts/nightly_runner.py --tests decision_head_backward_bf16 decision_head_backward --ci --require-executed
Local validation on 2026-11-07: nine BF16 tests and 138 combined BF16/FP32, ABI, numerical-resolution, provider-selection, ledger and documentation tests pass. Three selected standard-nightly suites pass without skips, including the pinned complete FP32 Clef head forward comparison. DSL policy and interface/allocation audits pass. Full repository nightly and BF16 full-head training were not run.
Correctness baseline, not a speed claim. These portable conversion adapters need no BF16 ISA but add conversion work and workspace. Benchmark a complete training step before claiming latency or peak-memory improvement. Complete BF16 head training, gradient accumulation precision, FP32 master weights/optimizer state, loss scaling policy and checkpoint dtypes remain separate contracts.
Next training evidence
The next completion gate is a small nonzero trainable head composition through the existing v8 workflow: verify every parameter gradient, demonstrate several native AdamW updates and loss reduction, prove frozen-weight identity, then save/reload and reproduce the updated scores. Expand this to the complete head once normalization, noncausal attention, projection and scoring derivatives are wired. The current primitive tests do not establish a trained decision model.
After FP32 head training, certify BF16 training and declarative LoRA independently. Quantized LoRA requires a selected frozen-base format with a tested backward-input path; inference Q4/INT8 kernels alone do not establish QLoRA. Keep model topology/trainability in graph declarations and optimizer arithmetic in the existing generated native workflow.
Backward audit and next training gate
Shared-consumer check on 2026-11-07: run_training_composition_v8.py completed both 2l_dense and 2l_gqa_tail with PASS. These existing generated decoder training certifications do not certify Clef backward execution.
Forward passes; head training remains NOT_TESTED. The full forward head is not yet linked to automatic backward execution. Extend cke.v8.compile and the existing generated-C optimizer,
checkpoint/resume and inference-export workflow. Do not add a second model executor.
| Head computation | Existing implementation | Missing contract/certification |
|---|---|---|
| Linear projections, biases | gemm_backward_f32.json maps fc2_backward_kernel; gemm_backward_bf16_mixed exists in gemm_kernels_bf16.c. | Bind checked head geometry and saved tensors; certify dX, dW and db, selective parameter gradients and accumulation. |
| LayerNorm | layernorm_backward_kernel in layernorm_kernels.c; BF16 variant in layernorm_kernels_bf16.c. | Cached-statistics training maps and native optimizer composition now pass; wire them into the complete head while preserving operation-local epsilon. |
| Erf GELU | Checked erf derivative now has FP32 VJP and generated-chain evidence; earlier gelu_backward_exact differentiates tanh. | Wire the erf provider into automatic head backward; BF16 remains separate. |
| Unmasked self/cross-attention | Checked FP32 noncausal dQ/dK/dV with ordered oracle, PyTorch autograd and native optimizer composition now pass. | Integrate the provider into the complete head. Masked and BF16 attention training are separate; existing causal providers are not substitutes. |
| Token-span pooling and last-token selection | Checked span VJP accumulates overlapping spans with generated-chain evidence. | Automatic span VJP and branch accumulation pass in the typed-input training composition; integrate final-row selection and the complete head. |
| Embedding/gather and residual branches | embedding_backward_f32.json, native BF16 embedding scatter, residual_add_backward_f32.json, gradient-accumulation providers. | Wire type-embedding gradients, indexed-owner scatter and broadcast reductions. Frozen output embeddings need no updates, while required hidden-feature gradient paths remain explicit. |
| Grouped softmax-weighted pooling | Checked options/query VJP covers both score and value paths, unequal and singleton groups. | Automatic VJP and shared-input branch accumulation pass in generated training; connect complete-head authoring and loss. |
| Cosine, pair features, scalar transforms | Indexed pair features and distinct normalized-dot/cosine FP32 VJPs now have generated branch-join evidence. | Automatic FP32 combined scoring now covers pair/cosine branches and learned scalar/gain gradients with 40 native updates. Complete released-head authoring and training remain uncertified. |
| Loss and optimizer | softmax_cross_entropy_loss.json and adamw_update_f32.json; existing trajectory/resume/export machinery. | Declare one label per question and loss reduction over unequal option counts, then verify every trainable parameter, frozen hashes, optimizer moments and restart equivalence. |
version/v8/scripts/run_training_workflow_v8.py:_apply_semantic_model currently admits
FP32, bias-free SwiGLU/RMSNorm decoder configurations at selected depths. Clef requires biased linear
operations, LayerNorm, erf GELU, branches, span pooling and grouped outputs. Removing that validation
alone would not implement their backward graph. Extend the responsible contracts while retaining the
existing decoder certification suite.
Start with FP32 head-only training on a small nonzero graph and frozen backbone features. Compare loss, feature gradients, every head parameter gradient, one AdamW update, a short deterministic trajectory, checkpoint/resume and inference reload. Then use released head geometry. BF16 kernel availability is not composed BF16 head-training evidence; BF16, adapters and quantized-base training remain separate later gates. End-to-end backbone decisions also need actual generated features and output embeddings, rather than synthetic ones.
Earlier option-summary milestone
build_clef_flash_option_summary_circuit.py extends the canonical ingress and both evidence-routing layers through grouped option pooling and option_summary_norm. Its single generated entry executes 43 operations. Question option counts partition consecutive routed option rows. Each question's projected vector supplies its pooling query. Normalization declares epsilon 1e-5 locally, independently of the enclosing model configuration.
The grouped provider validates positive counts and exact coverage, finite valid inputs, physical strides, capacities, alignment and disjoint writable buffers. Caller-owned scratch stages the result, so rejection does not publish partial output. Read-only buffers may alias each other. Padding is neither consumed nor written. Arithmetic follows the registered ordered FP64 reductions with FP32 intermediate storage; no backward capability is advertised.
Reproduction
python3 version/v8/decision/build_clef_flash_option_summary_circuit.py --check
python3 -m unittest tests.test_v8_grouped_weighted_rows tests.test_v8_clef_flash_option_summary -v
python3 scripts/nightly_runner.py --tests decision_grouped_weighted_rows decision_clef_flash_option_summary --ci --require-executed
Use a Python environment with NumPy; independent oracle tests additionally need PyTorch. The released-head comparison also needs safetensors and the dependencies imported by the pinned reference source. Enable that comparison with:
export CKE_CLEF_FLASH_HEAD_PATH=/path/to/joint_head.safetensors
export CKE_CLEF_FLASH_REFERENCE_SOURCE=/path/to/joint_schema_model.py
Both assets are hash-checked against tests/fixtures/decision/clef_flash/identity.json. Missing oracle packages or assets are explicit test skips. The reference test runs the pinned head's actual forward method and captures its summary normalization boundary; it does not inject reference evidence outputs into CKE. The host supplies synthetic hidden states, synthetic output-embedding rows, IDs and spans. Both implementations use the same inputs and released head weights. This is not a generated-backbone or complete-decision test.
On 2026-11-07, PyTorch 2.8.0 CPU comparisons passed for six tokens with option counts [2,1] and eight tokens with [1,2]. Maximum absolute errors were 1.90735e-6 / 1.43051e-6 for raw summaries and 1.31130e-6 / 1.19209e-6 for normalized summaries; normalized RMSE was 2.63926e-7 / 3.13428e-7. These are fixed compiled geometries, not proof of a dynamic serving interface. The declared pinned-prefix absolute regression ceiling is 1e-5.
Synthetic tests also cover reordered options within groups, A→B→A reuse, invalid counts before parity, undersized arenas, and rejection followed by successful execution. Model-neutral tests compare explicit FP64/FP32-stage arithmetic at O0/O3, including singleton groups, unequal groups and padded strides.
Related: kernel architecture, training capability ledger.