Deterministic Memory Layout

Why static memory offsets matter, and what they enable for monitoring, debugging, and mechanistic interpretability.

Why Deterministic Memory?

The Layout JSON: Your Memory Address Book

When you generate code with build_ir_v4.py, you also get a layout.json that maps every tensor to its exact memory location:

{
  "model": "Qwen2-0.5B",
  "layers": {
    "layer_15": {
      "attn_q": {"offset": "0x0A2400", "size": 1605632, "dtype": "bf16"},
      "attn_k": {"offset": "0x0A4800", "size": 1605632, "dtype": "bf16"},
      "attn_v": {"offset": "0x0A6C00", "size": 1605632, "dtype": "bf16"},
      "mlp_gate": {"offset": "0x0B2000", "size": 3211264, "dtype": "q4_k"}
    }
  }
}

This "address book" is the key to everything that follows. You can point to any tensor by name and know exactly where it lives in memory.

What This Enables

1. Zero-Overhead RDMA Monitoring

RDMA (Remote Direct Memory Access) supports one-sided operations. An observer can read memory from compute nodes without those nodes even knowing it happened—no interrupts, no CPU involvement, no overhead on training.

The observer laptop can sample any tensor at any rate (1Hz, 10Hz, 100Hz) without affecting training throughput. Traditional logging requires compute nodes to serialize and send data—that's overhead you can't avoid.

2. Live Tensor Inspector

┌─────────────────────────────────────────────────────────┐
│ Layer 15 Attention                                      │
├─────────────────────────────────────────────────────────┤
│ Q [0x0A2400 - 0x0A4800]  896×896 bf16  ████████░░ 2.1ms │
│ K [0x0A4800 - 0x0A6C00]  896×896 bf16  ██████████ 1.8ms │
│ V [0x0A6C00 - 0x0A9000]  896×896 bf16  ███░░░░░░░ 0.4ms │
│ Scores [0x0A9000]        896×896 f32   computing...     │
└─────────────────────────────────────────────────────────┘
    

3. Pipeline Parallel Visualization

Node 0 (Layers 0-7)    Node 1 (Layers 8-15)   Node 2 (Layers 16-23)
┌──────────────────┐   ┌──────────────────┐   ┌──────────────────┐
│ █ Micro-batch 0  │──▶│ █ Micro-batch 0  │──▶│ ░ waiting        │
│ █ Micro-batch 1  │   │ ░ Micro-batch 1  │   │                  │
│ ░ Micro-batch 2  │   │                  │   │                  │
└──────────────────┘   └──────────────────┘   └──────────────────┘
  RDMA: 12.4 GB/s        RDMA: 11.8 GB/s
    

4. Checkpoint Diff Tool

ck-diff checkpoint_step1000.bump checkpoint_step2000.bump

Layer 15 MLP Gate [0x0B2000]:
  Step 1000: mean=-0.0012, std=0.0234
  Step 2000: mean=-0.0018, std=0.0241
  Delta:     mean=-0.0006, std=+0.0007  ✓ healthy

Layer 23 Output [0x2A0000]:
  WARNING: std increased 3x - possible gradient explosion

5. Deterministic Replay Debugging

Mechanistic Interpretability

This is where deterministic memory becomes transformative. Mechanistic interpretability research aims to understand how neural networks actually work internally. The field struggles because frameworks hide everything behind abstractions.

Live Circuit Discovery

# While training is running, observe attention patterns
for layer in range(24):
    for head in range(14):
        attn = rdma_read(f"layer_{layer}_head_{head}_scores")
        if attn[query_pos, key_pos] > 0.8:
            print(f"Strong connection: L{layer}H{head}")

No hooks. No slowdown. Just read memory.

Activation Patching at Scale

# Causal intervention: "What happens if we zero this head?"
original = rdma_read(node=0, offset=LAYER_15_HEAD_7)
rdma_write(node=0, offset=LAYER_15_HEAD_7, data=zeros)
# Observe effect on output
rdma_write(node=0, offset=LAYER_15_HEAD_7, data=original)  # Restore

Neuron-Level Probing

Layer 15, Neuron 847:
  Activates strongly on: "capital cities", "country names"
  Offset: 0x0B2400 + 847*4
  Historical max: 12.4 (on token "Paris")
  Current: 0.02
  

Gradient Attribution

"Which weights most affected this prediction?"

∂Loss/∂W for every weight, at known offsets:
  layer_15_mlp_gate[0x0B2000]: 0.0023  ← high attribution
  layer_15_mlp_up[0x0B4000]:   0.0001  ← low
  layer_22_attn_o[0x1A2000]:   0.0089  ← highest!
  

Mechanistic First Pass (v6.6, Before v7 Training)

A practical first protocol should be small, repeatable, and comparable across models. Use the same prompts, token index, and context length each run.

Step 1: Run a Stable Repro Case

# Use one fixed model + prompt + token budget
python3 version/v6.6/scripts/ck_run_v6_6.py run \
  hf://Qwen/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q8_0.gguf \
  --context-len 1024 --max-tokens 32 --force-compile --force-convert \
  --prompt "Explain why matrix multiplication is associative."

Step 2: Probe Layer-Level Sanity (L0+)

python version/v6.6/test/test_layer_by_layer.py \
  --model ~/.cache/ck-engine-v6.6/models/Qwen--Qwen3-0.6B-GGUF \
  --token 25

This validates embedding and early-layer weight/normalization integrity without needing full decode analysis.

Step 3: Capture Stop-Point Activations

python version/v6.6/test/v6_6_comprehensive_debug.py \
  --model-dir ~/.cache/ck-engine-v6.6/models/Qwen--Qwen3-0.6B-GGUF \
  --token 25 --stop-at 5 --threads 1 --context-len 1024

Repeat with increasing --stop-at to walk the pipeline in order (embedding → norm → q/k/v proj → rope → attention → out_proj).

Step 4: Locate First Divergence

python version/v6.6/test/trace_divergence.py \
  --model ~/.cache/ck-engine-v6.6/models/Qwen--Qwen3-0.6B-GGUF \
  --token 25 --threads 1 --context-len 1024

Use this to identify the first failing op. That op is your intervention boundary for mechanistic analysis.

Step 5: Use IR Visualizer as the Analysis Surface

python version/v6.6/tools/open_ir_visualizer.py Qwen--Qwen3-0.6B-GGUF

• Interpretability tab: inspect math blocks and generated dump instrumentation.
• Dataflow tab: confirm producer/consumer chain for suspect ops.
• Profile tab: validate hotspot shifts are expected when changing kernels.

Attention Heads: Current and Next

Recommended Baseline Matrix (Before v7)

Comparison with Traditional Approaches

The Observer Architecture

A third machine (laptop, monitoring server) can observe training clusters via RDMA without affecting their performance:

# observer.py - runs on separate machine
class TrainingObserver:
    def __init__(self, nodes, layout_json):
        self.layout = load_layout(layout_json)  # Static offsets!
        self.rdma = RDMAConnection(nodes)

    def sample_gradients(self, layer):
        offset = self.layout[f"layer_{layer}"]["grad_offset"]
        size = self.layout[f"layer_{layer}"]["grad_size"]

        # One-sided read - compute node doesn't notice
        data = self.rdma.read(node=layer // 12, offset=offset, size=size)
        return np.frombuffer(data, dtype=np.float32)

    def check_for_nans(self):
        for layer in range(24):
            grad = self.sample_gradients(layer)
            if np.isnan(grad).any():
                alert(f"NaN detected in layer {layer}!")

The Philosophy

"We can't interpret what we can't see."

Dynamic frameworks hide computation behind layers of abstraction. The memory allocator decides where tensors live. Hooks and callbacks add overhead to observe what's happening.

C-Kernel-Engine takes the opposite approach: everything is explicit. The generated C code shows exactly what computation happens. The layout.json shows exactly where every tensor lives. RDMA lets you observe without disturbing.

This doesn't solve interpretability—the science is still hard. But it removes the infrastructure barriers that make interpretability research difficult. When you can see everything, you can start to understand everything.