Multi-State RWKV Online Memory

Mechanism-level experiments for comparing Dynamic Linear Attention (DLA) with RWKV-style online memory under controlled state and boundary policies.

HF checkpoint: xiaol/gemma-4-e4B-hybrid-rnn-mem-rwkv-fable5-gpt5.5-v1

Latest Trained-Model Result

The frozen Gemma4 + projected-slot outer-memory system now passes its preregistered native publisher-validation gate. The adapter is implemented in the repository's RWKV-MS-capable Delta-Mem runtime, but its active memory_readout_mode=projected_kv_slots bypasses the recurrent RWKV matrix scan. The decoder and all task routing rules were locked on publisher-TRAIN- derived development data before the validation split was opened. Evaluation used the identical frozen google/gemma-4-E4B-it comparator, greedy decoding, write-then-read online memory, the HF mirror, and four A100 GPUs.

All three tasks met the >=0.95 coverage floor, no task regressed, and two tasks improved. The scene result is the main effect: memory reduced false positives from 698 to 171, while true positives changed from 87 to 54 and false negatives from 84 to 117. Narrative gained five correct units. Attribution is preserved exactly by using the frozen-base candidate-likelihood scorer.

This is a system-level result for the locked decoder, not a claim that raw memory improves every task. Its fixed task policy is:

• attribution: frozen-base candidate likelihood;
• narrative: use memory only for the preregistered base=narration, memory=scene_description label pair;
• scene: use the projected-slot memory generation directly.

The reported scope is 238 rows. Attribution source row 0 was excluded in the protocol before the final run because it had already been touched by historical runtime diagnostics. Publisher test and Hard32 remain unopened.

Native Mechanism Accounting

The native validation, checkpoint-16, repair, and consistency-router results above and below establish a system-level gain from learned online outer memory. They do not establish that RWKV recurrence caused that gain. The frozen V9 adapter declares memory_backend=rwkv_ms, but its memory_readout_mode=projected_kv_slots forward branch writes and reads four content-addressed projected key/value slots per wrapped layer without invoking the recurrent _rwkv_ms_scan path. In the native one-shot benchmark, one last-valid-token proposal is written per layer; the four-slot capacity matters for sessions containing multiple write calls.

Accordingly, throughout the native-result sections:

• “online state” means the complete captured adapter state, whose active signal is the projected-slot key/value, occupancy, and surprise bundle;
• “memory gain” means a gain from that projected-slot Q/O adapter and its fixed task policy;
• “RWKV recurrence gain” is not claimed and requires a new matched experiment in which recurrent state materially contributes to the readout, with verified recurrent-state mutation and correct-state versus zero/donor/permuted- recurrent controls that hold projected slots fixed.

This correction changes the mechanism attribution, not the signed predictions, metrics, validation split discipline, or accepted system-level result. The CPU RWKV-7 studies, recurrent tau2 experiments, and recurrent GGUF runtime documented later are separate paths and are not reinterpreted by this note.

Reproducibility evidence:

• Locked publisher-validation protocol
• Signed validation decision
• Signed metrics and artifact hashes
• Validation runner and hash-bound analyzer
• Independent replication seeds R12 and R13 each passed all 52 sealed checks: R12 and R13

Next Recurrent Goal

The recurrent-only candidate remains rejected: despite finite nonzero output gradients in all 42 layers, its one-update BF16 correct-versus-zero and correct- versus-donor final-logit deltas were exactly 0.0. The replacement is now a carrier-controller hybrid. Projected KV slots remain the material retrieval carrier, while recurrent RWKV state modulates that carrier through a bounded readout that is exactly projected-only when recurrent state is zero.

A locked four-A100 screen tested residual, vector-gate, and scalar-gate equations at gains 0.03125, 0.0625, and 0.125. All nine candidates passed the fixed-carrier BF16 causal gates. The preregistered rule selected scalar_gate_g003125: the lowest gain and, at that gain, the smallest worst- rank perturbation from projected-only (1.3789 maximum absolute logit delta). Its minimum correct-versus-matched-donor recurrent delta was 1.3438.

The selected hybrid then passed a separately locked one-update BF16 calibration. All 42 recurrent output gradients were finite and nonzero, the global gradient norm was 0.2388, and both the full adapter and recurrent output weights changed. After the update, correct-versus-zero recurrent deltas ranged from 1.2812 to 1.9375 and correct-versus-donor deltas from 1.2969 to 1.8750 across the four ranks. Zero recurrent state remained exactly equal to the projected-only readout, with projected state byte-identical across interventions.

This established BF16-visible, trainable perturbation by RWKV state, but the subsequent three-seed native benchmark showed that perturbation was not the cause of the measured gain. The selected hybrid beat fresh projected-only controls by mean scene micro-F1 +0.00730; two of three seeds were nonnegative, and 8.48% of paired outputs changed. The per-seed gains were +0.02236, +0.00281, and -0.00326 for seeds 57, 58, and 59.

The recurrent causal gates failed. Mean correct-state minus zero-state micro-F1 was -0.00192, correct-state minus matched-donor was -0.00227, and correct- state minus layer-permuted was only +0.00052, below the locked +0.005 margin. Zero recurrent state exactly reproduced the projected-only bypass and the projected carrier remained byte-identical across interventions. The valid claim is therefore: the trained carrier-controller hybrid improves over a fresh projected-slot control on this authorized native benchmark, but correct RWKV recurrent state did not cause the improvement. The signed result status is native_benchmark_gain_without_recurrent_causal_pass; its canonical receipt is 7cd97cf939012c831bff96cdcc5fcfcf52ad3f626409d339814798cfa3c0d397.

The first information-bottleneck replacement, addressed_value, has now been tested. Projected keys supplied address/location information, projected values were excluded from the output path, and the complete bounded memory value came from the selected RWKV matrix. Zero recurrent state therefore produced exactly zero memory read, and zeroing every projected value was bit-exactly inert. The lowest-gain candidate (0.03125) passed the four-A100 structural screen: the minimum correct-versus-zero, donor, and layer-permuted maximum logit deltas were 1.375, 1.28125, and 1.3125. Its separately locked one-update calibration also passed, with finite nonzero recurrent-readout gradients in all 42 layers and global gradient norm 0.008994.

Eight causal-contrast updates completed, but their mean training margins did not stabilize in the desired direction: zero-minus-correct CE was -0.00467, donor-minus-correct was -0.00596, and layer-permuted-minus-correct was +0.00148. On the authorized 220-row publisher-TRAIN-derived native development partition, correct recurrent state scored 0.19287 micro-F1, versus 0.19135 for zero/empty, 0.18873 for a matched donor, and 0.19215 for layer-permuted recurrence. Every comparison was directionally positive, but the margins (+0.00152, +0.00414, and +0.00073) all missed the locked +0.005 threshold. The valid status is therefore addressed_value_native_gain_not_established, not recurrent native success.

The likely limitation was address/write misalignment rather than insufficient readout magnitude. A native write normally created one projected proposal in projected slot 0, while the recurrent scan wrote successive 128-token chunks to RWKV slots 0 through 3. The addressed read consequently tended to query RWKV slot 0 even when later chunks occupied other matrices.

The replacement chunk_addressed_value now creates one projected key for every RWKV slot touched by a write. Each key comes from the last contextual hidden state assigned to that exact recurrent slot; projected values are zero and remain excluded from the output path. An initial screen execution correctly failed its alignment audit because an inherited helper reset the write mode to the old residual hybrid. The corrected fresh execution placed two projected keys alongside the two nonempty recurrent chunks in every layer and rank. The lowest gain (0.03125) passed with minimum correct-versus-zero, donor, and layer-permuted maximum logit deltas of 1.28906, 1.21875, and 1.21875.

The separately locked one-update calibration also passed. All 42 recurrent readout gradients were finite and nonzero, the global gradient norm was 0.008295, recurrent output weights changed, and exact slot alignment, zero projected values, zero-state equivalence, and projected-value independence all survived the update.

Matched eight-update causal training then completed. Relative to the earlier single-address method, its mean donor-minus-correct CE improved from -0.00596 to -0.00078 and layer-permuted-minus-correct improved from +0.00148 to +0.00403; zero-minus-correct remained negative at -0.00471. These internal improvements did not transfer to generation. On the same authorized 220-row native development partition, correct recurrent state scored 0.19222 micro-F1, versus 0.19135 for zero/empty, 0.19350 for the matched donor, and 0.19314 for layer-permuted recurrence. The locked margins were therefore +0.00087, -0.00128, and -0.00092, all below +0.005. The signed status is chunk_addressed_value_native_gain_not_established.

Exact external chunk addressing was therefore not the way out. The next hybrid, recurrent_value, removed projected addressing from the read path entirely. RWKV's own cosine router scored all nonempty recurrent slots, and the bounded read was 0.03125 * tanh(read / rms(read)). Zeroing projected keys, values, occupancy, and surprise was bit-exactly inert, while zero, donor, and layer- permuted recurrent interventions were material on every A100 rank.

The one-update calibration passed with finite nonzero recurrent-output gradients in all 42 layers and global gradient norm 0.009567. The first causal-training preflight then correctly rejected 42 inactive projected_kv_key_proj tensors: they had remained in the optimizer even though the architecture could not read them. Protocol v2 froze exactly that read-inert family. Its fresh preflight had zero inactive trainables, and the fixed eight- update run completed with every recurrent gradient and carrier audit intact. The mean teacher-forced margins nevertheless remained negative: zero-minus-correct CE was -0.00406, donor-minus-correct was -0.00549, and layer-permuted-minus-correct was -0.00243.

The authorized 220-row native generation result also failed. Correct recurrent state scored 0.18967 micro-F1, versus 0.19135 for zero/empty, 0.19195 for the matched donor, and 0.18936 for layer-permuted recurrence. The locked margins were -0.00169, -0.00229, and +0.00031, all below +0.005; the signed status is recurrent_value_native_gain_not_established with receipt 1eecbb4a345e4bee390025089082757f7981e7147965e655d7c382952cf078b7.

The internal-router follow-up tested that hypothesis. A top-1 straight-through router became essentially one-hot, but its discrete forward route collapsed the useful optimization path. Top-2 routing preserved a differentiable mixture and passed a one-update contrastive calibration. Longer AdamW variants then failed deterministically at update 2 with non-finite per-row gradients, while an SPSA variant completed all eight updates but failed the held-out causal endpoint: zero-minus-correct, donor-minus-correct, and layer-permuted-minus-correct CE were -0.00102, +0.00454, and -0.00510.

Row-isolated gradient checks found one reproducible numerical culprit, source ordinal 1291. Filtering that whole row removed the update-2 instability. Positive-only training completed eight updates with 63/64 accepted rows but failed all three fresh endpoint margins (-0.00825, -0.00748, -0.00904). Filtered direct recurrent-value contrast training also completed, yet only the zero control was weakly positive: +0.000085 zero-minus-correct, versus -0.00545 donor-minus-correct and -0.00843 layer-permuted-minus-correct. This is evidence that the direct recurrent vector is not a reliable standalone material carrier, even after its numerical training failure is isolated.

The current candidate therefore restores projected KV as the material carrier and gives RWKV a narrower, causally testable role:

prompt hidden states
  |-- projected K/V writer --> four content-addressed carrier slots --|
  |-- RWKV-7 scan ----------> four recurrent 128-token states --------|-->
query hidden state --> temperature-16 detached top-2 RWKV router ------|
                                                                        |
projected_read * (1 + 0.03125 * clamp(cos(projected_read,
                                           recurrent_read), -1, 1))
  --> learned content gate --> Gemma attention output --> decoder

Thus RWKV is active, but it is not the retrieved value template: it writes the recurrent matrices, routes over them, produces recurrent_read, and controls a bounded scalar rescaling of the projected carrier. Zero recurrent state makes the cosine term zero and is exactly projected-only. Projected and recurrent states are separately captured so zero, matched-donor, and cyclic layer- permutation interventions can change RWKV while leaving every projected carrier byte-identical.

The selected scalar-agreement configuration uses temperature 16, top-2 routing, detached route scores, gain 0.03125, and content-gate initialization 0.25. Projected carrier and recurrent router tensors were frozen; 210 stable readout tensors were trained for eight row-isolated contrastive updates. All 64 rows were accepted. On a fresh 32-row teacher-forced endpoint it passed: zero-minus-correct CE was +0.29294, donor-minus-correct was +0.00746, and layer-permuted-minus-correct was +0.000474. This establishes held-out causal preference, not yet native generation gain.

The signed adapter config retained the constructor's earlier rwkv_ms_hybrid_mode=recurrent_value enum even though the signed training model audit ran scalar_gate; the candidate switch changed runtime attributes rather than the serialized config object. The locked generation evaluator discloses and hash-binds this mismatch, restores only that non-parameter runtime enum, and does not reinitialize any learned tensor.

The matched 220-row open native generation benchmark has now completed under exact zero-state/projected-bypass batch-shape controls. Correct recurrent state scored 0.18986 micro-F1, versus 0.18979 for zero/projected-only, 0.18710 for the matched donor, and 0.19149 for layer-permuted recurrence. The locked margins were therefore +0.000064, +0.00275, and -0.00163; all missed the required +0.005 gate. The valid signed status is scalar_agreement_native_gain_not_established, with receipt 9590428d136660e378b0b92ce79fcde5d46b7518314f9a629a04d8a9966c2e9e. Correct recurrence changed 9.55% of outputs relative to projected-only, but the changes did not improve native F1 or beat layer permutation, so no native RWKV recurrence gain is claimed.

The elementwise vector-FiLM follow-up then trained and passed its locked teacher-forced causal endpoint. Its zero-minus-correct, donor-minus-correct, and layer-permuted-minus-correct CE margins were +0.30025, +0.00393, and +0.01053. On the matched 220-row native generation benchmark it established a real gain over its own fixed carrier: correct recurrence scored 0.19426 micro-F1 versus 0.18763 for both zero recurrence and the explicit projected-only bypass, a +0.00663 margin above the locked +0.005 gate.

That is not yet a recurrent causal pass. The matched donor scored 0.19054, so correct-minus-donor was only +0.00372; cyclic layer permutation scored 0.19495, beating correct recurrence by 0.00069. Correct recurrence changed 10.00% of outputs relative to projected-only, and zero/projected-only outputs were exact while every projected carrier stayed byte-identical. The valid status is therefore vector_gate_native_gain_without_full_causal_pass, not native RWKV causal gain. Its signed receipt is 9fcbbd11ba502fdab77bee6c1177a5f5296cd4ff6ebecd0acdb3ce02b4cd10af.

The locked generation protocol contains two non-operative wording errors: one disclosure sentence says scalar_gate instead of the executed vector_gate, and another says generation sets the gain even though 0.125 was already serialized and only verified. The signed result carries both errata. Operative architecture fields, the fusion equation, training audit, evaluator assertions, and every prediction record consistently bind the executed mode to vector_gate; no learned tensor changed during restoration.

The result narrows the remaining problem from carrier gain to state specificity. The next bounded hybrid will test an alignment-gated residual:

a = clamp(cos(projected_read, recurrent_read), -1, 1)
projected_read
  + gain * rms(projected_read) * a
         * tanh(recurrent_read / rms(recurrent_read))

This keeps the successful projected carrier, retains exact projected-only identity for zero recurrence, and makes the directional correction depend on row- and layer-specific projected/recurrent agreement. A locked open-data screen must beat zero, donor, and layer permutation before training; no new native generation run is authorized otherwise. Publisher validation, publisher test, Hard32, and the unused strength holdout remain unopened and unauthorized.

Evidence: recurrent-only protocol, signed recurrent-only failure, hybrid screen protocol, signed hybrid screen, hybrid calibration protocol, signed hybrid calibration, hybrid screen runner, hybrid calibration runner, locked native benchmark protocol, signed native benchmark result, benchmark training runner, benchmark evaluation runner, hash-bound benchmark analyzer, addressed-value screen protocol, signed addressed-value screen, addressed-value calibration protocol, signed addressed-value calibration, causal-training protocol, signed causal-training endpoint, signed addressed-value native result, addressed-value runners, hash-bound addressed-value analyzer, chunk-addressed screen protocol, signed corrected chunk-addressed screen, chunk-addressed calibration protocol, signed chunk-addressed calibration, chunk-addressed causal-training protocol, signed chunk-addressed training endpoint, signed chunk-addressed native result, chunk-addressed evaluation runner, hash-bound chunk-addressed analyzer, focused chunk-addressed tests, recurrent-value screen protocol, signed recurrent-value screen, recurrent-value calibration protocol, signed recurrent-value calibration, corrected causal-training protocol, signed recurrent-value training endpoint, signed recurrent-value native result, recurrent-value evaluation runner, hash-bound recurrent-value analyzer, focused recurrent-value tests, sharp-router screen protocol, signed corrected sharp-router screen, top-2 contrast calibration protocol, signed top-2 calibration, SPSA causal-training protocol, signed SPSA failure, positive-only filtered protocol, signed positive-only filtered failure, filtered causal-contrast protocol, signed filtered causal failure, scalar-agreement training protocol, signed scalar-agreement causal endpoint, locked scalar native protocol, signed scalar native failure, scalar native evaluator, hash-bound scalar analyzer, focused scalar tests, vector-FiLM training protocol, signed vector-FiLM causal endpoint, locked vector-FiLM native protocol, signed vector-FiLM native result, vector-FiLM evaluator, hash-bound vector-FiLM analyzer, and focused vector-FiLM tests.

Post-Validation Mechanism Study

A preregistered 357-row publisher-TRAIN-derived study has now tested three online-state counterfactuals and 16 conservative scene routers. Correct state scored 0.2901 scene micro-F1, versus 0.1909 with zero state and 0.1923 after cyclically moving complete state bundles across the 42 wrapped layers. This is strong evidence that structured online state matters. The stricter causal gate did not pass, however: a different-gold, write-length-matched donor state scored 0.3001, beating the row-correct state by 0.0101. We therefore do not claim that row-specific episodic content uniquely causes the scene gain.

The router screen selected memory_plus_small_base_2: union frozen-base and memory boundaries only when the base predicts at most two. It improved fit micro-F1 from 0.2844 to 0.3190 and held-out development micro-F1 from 0.3093 to 0.3103. That held-out gain is real but only +0.0011, so this router is a future replication candidate and does not replace the accepted validation decoder.

A subsequent preregistered label-free state-retrieval study also failed its materiality gate. Four rules selected external states from a 1,443-row TRAIN-derived bank. Deterministic hash-random state won the 289-row fit screen, improving 0.2795 to 0.3093 micro-F1, but improved the exactly-once 68-row intervention holdout only from 0.3333 to 0.3352 (+0.0019, below the required +0.005). Semantic character-TF-IDF and token-length retrieval did not win. This points to weak generic state-induced regularization, not reliable semantic state retrieval.

Effective Q/O memory-strength calibration has now also completed. A signed excluded-row preflight proved that 0x, 0.5x, and 1x produce different outputs, then four A100s generated 1,136 candidates on the 284-row fit partition. Full strength remained best at 0.2915 micro-F1. The 0.5x and 0.75x candidates scored 0.2853; lower strengths scored 0.2195 and 0.2112. No intermediate strength passed the +0.005 gate, so no selection was written and the 73-row intervention holdout remains unopened. This rules out a fixed global amplitude as the next boundary, not memory learning itself.

Evidence:

• Locked causal/router protocol
• Signed causal/router decision
• Signed result and raw-artifact hashes
• Signed state-retrieval failure
• State-retrieval fit and holdout receipts
• Effective-strength protocol
• Signed effective-strength decision
• Effective-strength fit receipt

Contrast-Trained Scene Memory

The recommended training intervention has now succeeded on publisher-TRAIN- derived scene data. Only the 126 shared Q/O content-gate tensors were updated; all other adapter tensors remained bit-identical. Training used correct/no- state positives and different-gold, write-length-matched donor negatives on four A100 GPUs. A locked 64-row checkpoint probe selected step 16; step 32 was rejected because it no longer beat the donor control.

Checkpoint 16 then generalized to all 220 remaining open fit rows. On the combined 284-row fit partition it reached 0.3197 scene micro-F1, versus 0.2915 for frozen V9, 0.3058 for matched-donor state, and 0.1980 for zero state. The output changed on 25.7% of rows versus V9, and every preregistered coverage, native-gain, and causal-control gate passed. This is stronger than a generic state effect: the row-correct state now beats the matched donor.

This result does not replace the accepted publisher-validation number. It is a TRAIN-derived candidate result, and no validation predictions were used for checkpoint selection or analysis.

Evidence:

• Locked contrast-training protocol
• Signed training result
• Signed checkpoint-16 selection
• Locked full-fit progression
• Signed full-fit result

The subsequent multitask-preservation gate passed scene progression and exact attribution reuse, but failed its narrative comparator narrowly. Checkpoint 16 reached 0.5987 routed narrative unit accuracy on the untouched 114-row remainder, above frozen base (0.5847) but below V9's routed comparator (0.6007) by 0.0020. The complete signed failure is archived in the preservation result.

Hybrid Validation Replication

The failure is isolated to 12 narrative unit-level disagreements, so replacing V9 narrative behavior with checkpoint-16 output is rejected. The signed hybrid candidate keeps the proven task-wise contracts: frozen-base candidate likelihood for attribution, exact V9 routed output for narrative, and checkpoint-16 correct-state generation for scene. On the combined open TRAIN-derived fit rows this gives attribution accuracy 0.6966, V9 routed narrative accuracy 0.6007, and checkpoint-16 scene micro-F1 0.3197.

The TRAIN-derived hybrid gates passed without opening any protected split and authorized one separately preregistered publisher-validation replication. That replication generated every condition again from raw validation rows on four A100 GPUs; it did not read or reuse prior validation predictions.

The candidate improved two tasks over base, but the stricter training-gain gate failed: freshly regenerated V9 reached 0.2727 scene micro-F1, so checkpoint 16 was lower by 0.0016 instead of exceeding V9 by the required 0.005. The contrast-trained checkpoint therefore does not replace V9. The accepted V9 publisher-validation result at the top of this README remains authoritative. No publisher test, Hard32, or unused strength holdout evaluation is authorized.

Evidence:

• Locked hybrid protocol
• Signed hybrid result
• Hash-bound hybrid analyzer
• Locked fresh-validation protocol
• Signed fresh-validation failure
• Fresh-validation runner and analyzer

Cross-Fitted Scene Router

A subsequent TRAIN-only study tested a different method without reopening any protected split. Eight fixed V9/checkpoint-16 set-combination rules were selected independently inside five hash folds and scored only on each held-out fold. Four folds selected v9_if_subset_else_checkpoint; one selected raw checkpoint 16.

The cross-fitted router reached 0.3191 scene micro-F1, above frozen V9 (0.2915) but just below checkpoint 16 (0.3197) by 0.0006. It removed four false positives but also lost one true positive. The preregistered gate required a +0.005 gain over both inputs, so this method failed and no external replication is authorized. Simple set routing is therefore not the next boundary; further work must improve training robustness or expose calibrated token-level confidence while remaining publisher-TRAIN-derived.

Evidence:

• Locked cross-fit protocol
• Signed cross-fit failure
• Cross-fit analyzer

Checkpoint-Soup Failure

A second publisher-TRAIN-only study tested weight-space averaging instead of another output router. Seven convex recipes mixed only the 126 learned content- gate tensors from frozen V9 and contrast checkpoints 8, 16, and 32. The recipe bytes were signed and pushed before any new generation. Five hash folds then selected among those recipes with unchanged checkpoint 16 available as a fallback, using only the other four folds for each held-out decision.

The best single recipe, trajectory_centered (25% step 8, 50% step 16, 25% step 32), reached 0.3175 scene micro-F1 on the 220 post-probe fit rows. That is only +0.0021 over checkpoint 16 (0.3154), below the locked +0.005 requirement. Fold selection was also unstable: the five winners were trajectory_centered, s16_75_s32_25, checkpoint 16, trajectory_centered, and v9_25_s16_75. Their combined out-of-fold score was 0.3012, or -0.0142 versus checkpoint 16, despite remaining +0.0108 above V9 (0.2904). The method therefore failed and authorizes no external replication. Neither prediction-set routing nor convex checkpoint averaging is the next boundary; the next study must change training robustness itself.

Evidence:

• Locked checkpoint-soup protocol
• Signed candidate materialization
• Signed checkpoint-soup failure
• Four-GPU runner and cross-fit analyzer

Independent-Seed Robust Training Failure

The next publisher-TRAIN-only study changed training rather than selecting among outputs. Three independently hashed 256-row schedules trained the same 126 content-gate tensors on four A100 GPUs with global batch 16, half the original learning rate, and an explicit 0.995 post-step pull toward V9. All three runs were numerically clean and tightly matched in endpoint delta norm (0.1001, 0.1039, and 0.0973). Their pairwise delta cosines were 0.611-0.659. The only preregistered candidate was the equal mean of the three signed V9-relative deltas, fixed and pushed before generation.

That candidate reached 0.3059 scene micro-F1 on the same 220 open fit rows. It remained above V9 (0.2904) by +0.0155, but fell below checkpoint 16 (0.3154) by -0.0094. Relative to checkpoint 16 it gained one true positive but added 21 false positives, so the locked +0.005 gate failed. The averaged delta was only 0.0873 from V9 versus 0.1847 for checkpoint 16 and had cosine 0.566 with checkpoint 16's delta. The lower-rate V9-centered ensemble therefore stabilized an underpowered direction rather than preserving the single checkpoint's precision. It authorizes no external replication. A next training study should anchor small independent residual updates at checkpoint 16 instead of pulling every run back toward V9.

Evidence:

• Locked seed-ensemble protocol
• Signed seed-delta materialization
• Signed TRAIN-only failure
• Four-GPU trainer, evaluation runner, and analyzer

Checkpoint-16 Residual Training Failure

A follow-up publisher-TRAIN-only study preserved checkpoint 16 as the anchor instead of restarting from V9. Three four-GPU runs used disjoint sets of 128 previously unused rows, eight global-batch-16 updates, learning rate 2.5e-5, and a 0.995 post-step pull toward checkpoint 16. The runs were numerically clean, changed no frozen parameters, and ended only 0.0251, 0.0269, and 0.0258 from the anchor. Their residual directions were weakly aligned, however: pairwise cosines were 0.049-0.189. The locked equal residual mean therefore had norm 0.0169 and was fixed and pushed before generation.

The candidate reached 0.3129 scene micro-F1 on the same 220 open fit rows. It preserved checkpoint 16's 79 true positives and 161 false negatives, but added four false positives, reducing micro-F1 by -0.0025 from checkpoint 16 (0.3154). It still remained +0.0225 above V9 (0.2904) and changed 6.36% of checkpoint-16 outputs, but failed the preregistered +0.005 improvement gate. It authorizes no external replication, and publisher validation, test, Hard32, and the unused 73-row holdout remain unopened. Repeated endpoint averaging is no longer the useful boundary: the next training intervention must directly suppress false-positive scene labels while preserving checkpoint 16's true positives.

Evidence:

• Locked checkpoint-16 residual protocol
• Signed residual materialization
• Signed TRAIN-only failure
• Four-GPU residual trainer, evaluation runner, and analyzer

Precision-Unlikelihood Training Failure

The next publisher-TRAIN-only intervention started from checkpoint 16 and used 256 previously untouched eligible rows. One four-A100 run made 16 global-batch- 16 updates at learning rate 1.5e-5, with a 0.995 post-step pull toward the starting checkpoint. Each row combined unit-weight gold teacher-forced CE with weight-0.5 unlikelihood on only the decimal token or tokens of one inserted false boundary under the same correct online state. JSON syntax was never a negative target. Of the 256 negatives, 199 inserted boundary 1; the loss penalized 270 false-boundary digit tokens in total. The run changed only the 126 content-gate tensors, ended 0.0499 L2 from checkpoint 16, and was fixed and pushed before generation.

The candidate lost 0.0219 micro-F1 from checkpoint 16 and missed the locked gain over V9 by 0.0019. It changed 23 of 220 outputs: nine additions, seven removals, and seven substitutions. Those changes added 30 false boundaries while removing only 10, lost four true boundaries while adding one, and included one unstable row that added all 15 boundaries from 2 through 16. Boundary 1 itself was not calibrated: five false instances were removed but five new false instances appeared, while two true instances were lost and one was added. Teacher-forced digit-only unlikelihood therefore did not transfer to greedy-set precision; combined gold CE also failed to preserve the local checkpoint-16 decision surface. The endpoint is archived without external replication. Publisher validation, publisher test, Hard32, and the unused 73-row strength holdout remain unopened.

Evidence:

• Locked precision-unlikelihood protocol
• Signed training endpoint
• Signed materialization
• Signed TRAIN-only failure
• Four-GPU trainer, evaluation runner, and analyzer

On-Policy First-Divergence Repair Failure

A materially different publisher-TRAIN-only run mined 96 previously untouched eligible rows from frozen checkpoint 16 before optimization. It updated only the 126 content-gate tensors for six global-batch-16 steps on four A100 GPUs. Rows with a generated false-positive boundary contributed one pairwise loss at the first divergence from gold; there was no full-sequence gold CE and no synthetic-negative unlikelihood. Learning rate 5e-6, gradient clipping at 0.05, and a 0.99 checkpoint-relative retention kept the final move to only 0.00586 L2. The run found 53 actionable repairs representing 62 false boundaries, changed no non-gate state, and was fixed and pushed before generation.

The endpoint changed 14 of 220 outputs and improved recall, gaining three true boundaries without losing any. It nevertheless added 18 net false positives and missed checkpoint 16 by 0.00119 micro-F1. One row caused almost the whole failure: source row 187 changed [1] into every boundary [1, ..., 16], adding 15 false positives. Excluding that diagnostic row, the endpoint would score 0.3235, but that is not the benchmark result and no row is excluded from the signed verdict. First-divergence repair found a useful local direction but did not constrain the rest of the generated trajectory. The next intervention must preserve checkpoint-16 continuation behavior while applying repairs, rather than adding stronger global precision pressure. No protected split is authorized or opened.

Evidence:

• Locked on-policy repair protocol
• Signed training endpoint
• Signed materialization
• Signed TRAIN-only failure
• Four-GPU trainer, evaluation runner, and analyzer

Adaptive Gold-Suffix Repair Failure

A follow-up development study reused the same 96 publisher-TRAIN rows after observing the on-policy result, so it is explicitly adaptive and cannot authorize external replication or protected evaluation. It retained the first-divergence pairwise objective and added weight-0.25 teacher-forced CE on as many as four gold tokens beginning at the divergence. Six four-A100 updates used learning rate 2.5e-6, clipping at 0.025, and 0.995 checkpoint-relative retention. The final content-gate move was only 0.00302 L2 from checkpoint 16 and was fixed before generation.

The endpoint changed 12 of 220 outputs, for a net gain of one true positive but 15 false positives. It trailed checkpoint 16 by 0.00589 micro-F1 while remaining 0.01910 above V9. Crucially, source row 187 still changed [1] into [1, ..., 16]: gold-suffix CE trained the corrected branch, but did not constrain the generated continuation after the model still selected the wrong [1 branch. The signed result receipt is 75bff5880533653f8748bffb5dda8386335a7bee7df668fbfa2c853555caff02. Publisher validation, publisher test, Hard32, and the unused 73-row holdout remain sealed.

Evidence:

• Locked suffix-repair protocol
• Signed training endpoint
• Signed materialization
• Signed adaptive TRAIN-only failure
• Four-GPU trainer, evaluation runner, and analyzer

Adaptive Dual-Path Continuation Repair Failure

The next adaptive study kept the original first-divergence pairwise correction but added a second forward/backward path for each actionable row. That path conditioned on the checkpoint-16 prefix through the original wrong token and distilled as many as three subsequent checkpoint tokens with weight 0.25. This directly constrained behavior after a wrong branch instead of training only the corrected branch. The same reused 96 TRAIN rows produced 53 repairs over six four-A100 updates at learning rate 5e-6, clipping 0.05, and retention 0.99. The final content-gate move was 0.00591 L2.

The intervention solved its targeted failure: source row 187 remained the checkpoint output [1] instead of expanding to [1, ..., 16]. It changed only seven of 220 outputs and removed false boundaries on two rows, including an exact correction from [1, 10] to [10] on source row 190. However, it added six false boundaries on five other rows, changed no true-positive count, and ended four false positives above checkpoint 16. Micro-F1 therefore remained 0.00250 below checkpoint 16, while exceeding V9 by 0.02249. Wrong-branch continuation distillation is a useful stability constraint, but at this weight it suppressed the on-policy endpoint's three-TP recall gain. The signed result receipt is 08d8359f4aaeb7f9e5367c0999abd6ce2e74a4cb73dfd9ce9b7a32c3887e5f58. No external replication or protected evaluation is authorized.

Evidence:

• Locked dual-path protocol
• Signed training endpoint
• Signed materialization
• Signed adaptive TRAIN-only failure
• Four-GPU trainer, evaluation runner, and analyzer

Adaptive Convex Repair-Bridge Failure

The next TRAIN-only study tested whether the recall and continuation behavior could be separated in weight space without another training run. It materialized exact convex gate-only bridges between the frozen on-policy and dual-path endpoints at 25%, 50%, and 75% on-policy weight. All three recipes, source artifacts, 126 tensors, and 108,906 values were hash-bound before four A100s generated any outputs. A fresh five-fold cross-fit selected among the three bridges and unchanged checkpoint 16 using only the other four folds.

The interpolation exposed a discrete generation threshold rather than a smooth precision/recall frontier. Source row 187 stayed at [1] with 25% on-policy weight, but both 50% and 75% abruptly regenerated every boundary [1, ..., 16]. The 75% bridge recovered two of the on-policy endpoint's three true positives, yet its 14 extra false positives left it 0.00202 micro-F1 below checkpoint 16. Cross-fit selected a learned bridge in only one of five folds and trailed checkpoint 16 by 0.00857; the signed study therefore failed. The result rules out global linear interpolation as the next step and motivates conditional inference on the small set of rows where the endpoints disagree. Its receipt is 98eb982c239dd7e845e597d7294aef5aeeab3e18c906505205d0373521f44717. No protected split was opened or authorized.

Evidence:

• Locked bridge protocol
• Signed materialization
• Four-GPU raw generations
• Signed adaptive TRAIN-only failure
• Materializer, evaluation runner, and cross-fit analyzer

Adaptive Consistency-Router Success

The first method to clear the locked +0.005 improvement threshold on the native TRAIN-derived scene benchmark is a conservative dual-pass consistency router rather than another globally trained endpoint. It compares checkpoint 16 with the recall-heavier 75% on-policy bridge and accepts the proposal only when it is a strict subset of the checkpoint prediction, or when checkpoint abstains and the proposal is a single boundary. The rule never sees gold at inference. It was designed after inspecting the bridge failure, so it is explicitly post-hoc adaptive evidence, not an independent benchmark claim.

A fresh five-fold cross-fit selected the combined rule on all five fit partitions, so its out-of-fold result equals the aggregate result. The router changed six of 220 outputs: two singleton proposals repaired missed true boundaries on rows 89 and 178, while four strict-subset proposals removed one false boundary each on rows 190, 305, 321, and 354. This gives +2 TP, -4 FP, and +0.00928 micro-F1 over checkpoint 16. It also avoids the row-187 cascade because the 16-boundary proposal is neither a strict subset nor a singleton after abstention.

This is a real success on the open publisher-TRAIN-derived native development benchmark, but it does not yet establish generalization: the rule family was created after observing those rows, requires two generation passes, and has not been tested on any protected split. The next boundary is a pre-registered replication on genuinely new data, followed by distilling the decision into a single-pass confidence head only if replication succeeds. The signed receipt is c7f2d6af754c843cea5abbc8cf96415182d4ae97838586fdbea8d46a15ba049a.

Evidence:

• Locked adaptive router protocol
• Signed TRAIN-only success
• Router and cross-fit analyzer

This repository starts from the Log-Linear Attention codebase and adds a CPU-only proof of concept in dla_poc.py. It reproduces the core DLA mechanism from arXiv 2606.10650 and adds HRM-Text-inspired memory baselines:

• rwkv_mem(delta_rule): single online delta-rule associative memory.
• rwkv_mem(rwkv7): single read-before-write RWKV-7 state.
• rwkv_mem(rwkv7 multi-state): same RWKV-7 state update, but one state per adaptive memory block.
• State-only ablation: fixes the exact same boundaries for linear/DLA states and RWKV-7 states, so the comparison isolates the state update.

Quick Start

python3.12 -m venv .venv
PATH="$PWD/.venv/bin:$PATH" bash run.sh

If the environment is already set up:

.venv/bin/python dla_poc.py

Outputs are written to:

EVAL.md
.openresearch/artifacts/dla_summary.json
.openresearch/artifacts/dla_trials.jsonl
.openresearch/artifacts/dla_comparison.png
.openresearch/artifacts/run_log.txt

Mechanism-Level Result

The main DLA reproduction still passes:

• DLA lowers the Theorem 3.1 deviation bound in every tested config.
• DLA beats fixed Log-Linear blocking on needle recall at matched state count.
• The repo Log-Linear attention smoke test passes on CPU.

Mechanism recall comparison:

State-update-only comparison, with identical boundaries for both states:

This table fixes the exact same token blocks for both methods. linear/DLA state uses the standard block sum sum k_t v_t^T; RWKV-7 state uses the RWKV-7 recurrence inside each same block. Therefore each row compares state update/readout only, not boundary quality.

Interpretation:

• With perfect or near-perfect boundaries, linear/DLA state and RWKV-7 state tie on this synthetic recall task.
• When boundaries are fixed, noisy, or compressed to low K, RWKV-7 state is more robust in this task.
• DLA's main advantage is adaptive boundary/state allocation; RWKV-7's advantage appears in the state update when boundaries are held fixed and imperfect.

Full tables are in EVAL.md.

What Is Compared

dla_poc.py runs four groups of checks.

1. Codebase smoke test - Loads the original Log-Linear Attention pure PyTorch path directly. - Avoids CUDA-only Triton/Mamba dependencies.

2. DLA deviation-bound check - Implements Algorithm 1: information-aware dynamic state merging. - Implements Algorithm 2: capacity-bounded adjacent state merging. - Compares DLA blocks against fixed contiguous blocks at matched state count.

3. Needle associative recall - Uses synthetic rare needle tokens mixed with redundant filler tokens. - Compares fixed blocks, DLA blocks, delta-rule memory, RWKV-7 memory, and multi-state RWKV-7 memory.

4. State-update-only ablation - Uses the same block boundaries for both linear/DLA and RWKV-7 state update. - Boundary policies: oracle, dla, fixed, noisy_dla, low_k_dla. - This isolates whether the state update/readout is stronger, independent of boundary selection.

Scope

The top-level DLA comparison is a training-free mechanism reproduction. It does not reproduce 50B-token pretraining or trained HRM-Text checkpoints. The separate Gemma4 online-memory experiments documented above and under experiments/rethinking_rwkv_ms_gemma/ include both projected-slot and recurrent RWKV-MS readouts; the native validated V9 line is the projected-slot variant described in “Native Mechanism Accounting.”

The HRM/RWKV baselines are self-contained ports of the memory recurrence ideas, not full imports of HRM-Text:

• rwkv_mem(delta_rule) follows the read-before-write delta-rule associative state from HRM-Text's models/rwkv_memory.py.
• rwkv_mem(rwkv7) follows the latest read-before-write RWKV-7 recurrence from HRM-Text's models/rwkv7.py, specialized to the synthetic key/value stream.

Repository Layout

dla_poc.py                         # Main reproduction and comparisons
run.sh                             # CPU dependency install + run
EVAL.md                            # Generated report from latest run
.openresearch/artifacts/           # JSONL, JSON, figure, run log
hattention/                        # Log-Linear Attention implementation used for smoke test
figs/                              # Original figure asset
deltamem/                          # bundled patched HF online-memory runtime
integrations/delta_mem_rwkv_ms/    # launchers, docs, GGUF tools, optional upstream patch
integrations/delta_mem_rwkv_ms/gguf/ # GGUF sidecar, fixture, and parity helpers

HOLA Hippocampus on RWKV-7 Multi-State

experiments/hola_hippocampus/ replaces the neocortex of HOLA (arXiv 2607.02303, semiparametric memory = compressive state + bounded exact-KV cache) with this repo's read-before-write RWKV-7 multi-state and re-tests HOLA's design claims on the state-only ablation grid. The mapping is exact: for unit keys the RWKV-7 correction term makes the update a delta rule, so HOLA's surprise score beta*||e|| becomes the write magnitude m_t = ||Delta_t||_F already computed by the recurrence.

Result summary (5 seeds; full tables in experiments/hola_hippocampus/REPORT.md):

• The weakest state-only cell (16 needles, low_k_dla, 0.669 above) rises to 0.880 with a 16-slot surprise cache; a matched recency cache stays at 0.665.
• HOLA's two claims reproduce on RWKV-7: recency caching is dead weight for far needles, and a flat softmax read (0.83*cos) equals no cache at all.
• One correction was required: raw surprise admission fails with an untrained constant gate; an online CLS-style consolidation rule (demote cache entries whose key the state later predicts well) plus a read-confidence gate makes the cache strictly non-harmful. Hypothesis ledger and run provenance live in .keel/.

Run: .venv/bin/python experiments/hola_hippocampus/hola_rwkv_ms.py

Delta-Mem RWKV-MS Online Memory

The practical RWKV-MS online-memory integration is self-contained in this repository. The patched Python runtime is bundled at top-level deltamem/, so normal Qwen/Gemma HF training and inference do not require another delta-Mem checkout. integrations/delta_mem_rwkv_ms/ contains HF inference and verified manual training-smoke entry points, a matched delta-rule/RWKV-MS launcher, GGUF tools, and an optional upstream patch export. The runtime supports Qwen3, Qwen3.5/Qwen3.6, SmolLM3, and Gemma4 text attention; for google/gemma-4-E4B-it it wraps the non-KV-shared attention layers and skips the KV-shared tail layers.

Fresh RWKV-MS configs use semantics v2: FP32 recurrent matrices, bounded per-head write sources, RWKV decay without a second lambda decay, cosine slot routing, and a bias-free empty-state readout. Checkpoints without an explicit rwkv_ms_semantics_version load as legacy v1 and must not be resumed as v2; start a fresh run with --rwkv-ms-semantics-version 2.

Transformers exposes Qwen3.6 as qwen3_5. Its 64-layer hybrid stack has 16 full-attention layers at physical indices 3,7,11,...,63; the other 48 Gated DeltaNet layers are not wrapped. Use layer 3 for a smoke run or 3,7,11,15,19,23 for the six early eligible layers. This Qwen path is the HF integration and is separate from the Gemma-only GGUF sidecar runtime.

The bundled deltamem/ package provides the wrapper/session machinery: attaching online-memory modules to a Transformers model, loading delta_mem_adapter.pt, keeping RWKV-MS state synchronized with the KV cache, and applying the chat template. The optional integrations/delta_mem_rwkv_ms/delta_mem_rwkv_ms.patch exports these changes for upstream delta-Mem revision 5cd5d9153c7f408764728d953565201e198c39e2; it is not needed for normal use of this repository. See bundled runtime provenance for the source snapshot and local integration revision.

For HF workflows, install the bundled package from the repository root before running the commands below:

pip install -r requirements.txt
pip install -e .

Gemma tau2 status

The active Gemma + RWKV-MS tau2 recipe is documented in GEMMA_RWKV_MS_TAU2_TRAINING_PLAN_V2.md. For reproducibility, the benchmark artifacts record this historical source integration commit (it is not a current external runtime dependency):

bec8330 Add RWKV-MS memory backend for Gemma tau2

Current best learned no-rule online-memory checkpoint:

xiaol/gemma-4-e4B-hybrid-rnn-mem-rwkv-fable5-gpt5.5-v1

Local source checkpoint:

/run/media/xiaol/B214449214445C0B/delta_mem_outputs/gemma_rwkv_ms_tau2/v2ruleplanner_mobile_focusedtools_turns_formatrefresh_continue200_len192_layers0_5_qo_r8/checkpoints/step-100

The table below keeps learned online-memory runs separate from rule-assisted diagnostic runs. "No-rule" means no eval-time --mobile-data-rule-planner and no parser format-repair patch.

Release framing: "One can have both the fish and the bear's paw." The base Gemma checkpoint remains frozen to preserve original behavior, while the learned RWKV-MS path adds a small recurrent memory surface that can be adapted to local domain data.

Memory-path size from saved checkpoints:

Local training-cost notes:

• Experiments were local CUDA bf16 runs on an RTX 4090 24 GB setup.
• Generated mobile-data action SFT used 3,519 turn rows for 656 optimizer steps.
• Format-refresh continuation used 5,027 turn rows for 200 optimizer steps.
• Exact wall time and VRAM vary with local hardware, sequence length, layer count, rank, cache location, and fragmentation; adapt the frozen-base online-memory recipe to your own data.

Status interpretation:

• The original tau2 data was the problem: the 82-row run trained for 656 optimizer steps and its loss moved, but the benchmark collapsed to 1/20.
• Generated mobile-data action SFT transfers better, and the 6-layer shallow online-memory path is the current useful capacity point.
• The 200-step format-refresh continuation overtrains relative to its step-100 checkpoint, so checkpoint selection matters.
• The eval-time rule planner / float-format fix is excluded from the comparison table because it is benchmark-specific control logic, not model behavior.
• The next benchmark should run the step-100 checkpoint on at least 50 tasks, preferably the full telecom split, before treating 14/20 as robust.

Recommended HF online-memory inference command:

python integrations/delta_mem_rwkv_ms/inference.py \
  --memory-repo xiaol/gemma-4-e4B-hybrid-rnn-mem-rwkv-fable5-gpt5.5-v1 \
  --base-model google/gemma-4-E4B-it \
  --device cuda:0 \
  --dtype bfloat16 \
  --attn-implementation sdpa

Gemma4 GGUF First Step

A base Gemma4 E4B GGUF has been downloaded for llama.cpp testing on the 2 TB SSD:

/run/media/xiaol/B214449214445C0B/models/gguf/gemma-4-E4B-it/gemma-4-E4B-it-Q8_0.gguf
/run/media/xiaol/B214449214445C0B/models/gguf/gemma-4-E4B-it/mmproj-gemma-4-E4B-it-Q8_0.gguf
/run/media/xiaol/B214449214445C0B/models/gguf/gemma-4-E4B-it/gemma-4-E4B-it-rwkv-ms-memory.gguf

The first two files are normal base-model inference artifacts. The RWKV-MS memory file is a GGUF sidecar containing the adapter tensors and metadata. The local llama.cpp branch can now consume that sidecar in an experimental Gemma4 runtime path: model load owns the sidecar tensors in CPU buffers, the Gemma4 graph applies RWKV-MS q,o deltas on target layers 0-5, and a mutable RWKV-MS state buffer is updated during prompt/generation scans. The current runtime is intentionally constrained to one sequence. The server/UI path keeps physical microbatches serial (-ub 1) for the best-tested state behavior; the CLI graph can build experimental graph-unrolled multi-token prompt scans, but that path still needs stronger state-level parity coverage before it should be treated as production-ready. See GGUF_EXTERNAL_MEMORY_FEASIBILITY.md and integrations/delta_mem_rwkv_ms/GGUF_PORT_PLAN.md. At llama.cpp model load time, the sidecar path now performs semantic validation before runtime use: it verifies delta_mem.base_gguf_sha256 against the exact loaded base GGUF file, then rejects unsupported num_state_heads != 1, duplicate compact tensor names, missing required tensors, and wrong ggml-order tensor shapes.

Patched llama.cpp fork:

https://github.com/xiaol/llama.cpp-online-memory
branch: main
commit: 85da0c63b Add Gemma4 RWKV-MS GGUF sidecar runtime
base upstream: ggml-org/llama.cpp 1ec44d1

Current sidecar identity:

sha256: 0c646a776b5b12c9d3657ffd2e5e581be1eb46e858f1f404afeaa7077c02974e
bound base GGUF sha256: fb8f0c032de00b18c710824af3c7e5777c71e5fb60b13f13575f0a9e92ddecd0
size: 1,663,840 bytes
tensor name format: compact_with_source_name_manifest
tensors: 186 BF16

Start a recent llama.cpp server:

LLAMA_SERVER_BIN=/run/media/xiaol/B214449214445C0B/tools/llama.cpp/build-cuda/bin/llama-server \
LLAMA_REASONING=off \
bash tools/llama_server_gemma4.sh

For the experimental RWKV-MS sidecar runtime through llama-server, use the patched llama.cpp build and the constrained sidecar mode:

mkdir -p .openresearch/artifacts/gguf_ui
LLAMA_SERVER_BIN=/run/media/xiaol/B214449214445C0B/tools/llama.cpp/build-cuda/bin/llama-server \
LLAMA_PORT=18083 \
LLAMA_RWKV_MS=1 \
LLAMA_REASONING=off \
bash tools/llama_server_gemma4.sh 2>&1 | tee .openresearch/artifacts/gguf_ui/llama_server_rwkv_ms.log

The helper sets --rwkv-ms-sidecar, --batch-size 2, --ubatch-size 1, --parallel 1, disables continuous batching/context shift/prompt-cache reuse, disables server prompt-cache RAM and context checkpoints, enables a slot-save directory for manual slot 0 save/restore, and uses text-only mode for the current one-sequence runtime. The patched llama.cpp context rejects sidecar runs with more than one sequence. The server/helper keep --ubatch-size 1, reject speculative decoding, and preflight unsafe slot/cache/batch overrides before starting llama-server; model load also rejects malformed or unsupported sidecars before any RWKV-MS graph consumes their tensors. A sidecar exported for a different base GGUF now fails model load with a hash mismatch instead of running against the wrong weights.

For the best-tested experimental runtime path, pass the sidecar and use serial physical microbatches:

/run/media/xiaol/B214449214445C0B/tools/llama.cpp/build-cuda/bin/llama-completion \
  -m /run/media/xiaol/B214449214445C0B/models/gguf/gemma-4-E4B-it/gemma-4-E4B-it-Q8_0.gguf \
  --rwkv-ms-sidecar /run/media/xiaol/B214449214445C0B/models/gguf/gemma-4-E4B-it/gemma-4-E4B-it-rwkv-ms-memory.gguf \
  -p "Hi" -n 32 -c 96 -b 2 -ub 1 -ngl 99 --no-warmup --no-display-prompt \
  --no-perf -no-cnv -s 123 --temp 0 --top-k 1

With the same seed and greedy sampling, the base and sidecar paths now diverge. The sidecar path produced ! I'm excited to chat with you. What's on your mind today? ..., while the base path continued ! I'm excited to chat with you. I'm here to help .... Treat this as a smoke signal consistent with the sidecar path; confirm runtime use with server logs and the reference-trace health check.

The local CUDA build is from the online-memory fork commit 85da0c63b, based on upstream llama.cpp 1ec44d1, and detects the RTX 4090 as CUDA0. CUDA 13.1 plus GCC 15 needed a local header shim during build; the resulting binary is under the SSD tool directory above.

Then launch the local testing UI:

python3.12 -m venv .venv-ui
.venv-ui/bin/pip install -r requirements-ui.txt
LLAMA_BASE_URL=http://127.0.0.1:18083/v1 \
LLAMA_RWKV_MS=1 \
LLAMA_MODEL=gemma-4-e4b-it-rwkv-ms-q8 \
GGUF_RWKV_MS_SIDECAR_PATH=/run/media/xiaol/B214449214445C0B/models/gguf/gemma-4-E4B-it/gemma-4-E4B-it-rwkv-ms-memory.gguf \
LLAMA_SERVER_LOG=.openresearch/artifacts/gguf_ui/llama_server_rwkv_ms.log \
GGUF_RWKV_MS_HEALTH_OUTPUT=.openresearch/artifacts/gguf_ui/rwkv_ms_runtime_health.json \
GGUF_UI_REQUIRE_RWKV_MS_HEALTH=1 \
GGUF_UI_PORT=7861 \
.venv-ui/bin/python tools/gemma_gguf_ui.py

Before comparing prompts, verify that the endpoint is really the patched sidecar runtime. The UI exposes the same check through its RWKV-MS runtime button, writes the health file, and blocks sidecar chat/trace comparison while the selected endpoint/model/sidecar/log do not match a recent successful check.

.venv-ui/bin/python tools/check_rwkv_ms_gguf_runtime.py \
  --base-url http://127.0.0.1:18083/v1 \
  --server-log .openresearch/artifacts/gguf_ui/llama_server_rwkv_ms.log \
  --output .openresearch/artifacts/gguf_ui/rwkv_ms_runtime_health.json

The check requires the server log because API output alone cannot prove that llama.cpp loaded the RWKV-MS sidecar. It verifies model listing, a chat smoke request, the saved reference trace, slot 0 save/restore with exact-prefix continuation, corrupted slot restore rejection, and log evidence for RWKV-MS activation, one server slot, disabled prompt cache, disabled context checkpoints, and exact-prefix slot reuse. The sidecar server also rejects speculative decoding options.

For repeatable prompt checks against the same server:

.venv-ui/bin/python tools/eval_gguf_prompts.py configs/gguf_rwkv_ms_prompt_suite.jsonl \
  --base-url http://127.0.0.1:18083/v1 \
  --model gemma-4-e4b-it-rwkv-ms-q8 \
  --rwkv-ms \
  --temperature 0 \
  --seed 42

For the RWKV-MS side of the future port, inspect the PyTorch memory checkpoint into a tensor/config manifest:

.venv/bin/python integrations/delta_mem_rwkv_ms/gguf/inspect_memory_checkpoint.py \
  --memory-dir /run/media/xiaol/B214449214445C0B/models/delta_mem/gemma-4-e4B-hybrid-rnn-mem-rwkv-fable5-gpt5.5-v1 \
  --output .openresearch/artifacts/gguf_memory_manifest.json

To regenerate and validate the GGUF memory sidecar:

.venv/bin/python integrations/delta_mem_rwkv_ms/gguf/export_memory_gguf.py \
  --manifest-output .openresearch/artifacts/rwkv_ms_memory_sidecar_manifest.json
.venv/bin/python integrations/delta_mem_rwkv_ms/gguf/inspect_memory_gguf.py \
  --memory-dir /run/media/xiaol/B214449214445C0B/models/delta_mem/gemma-4-e4B-hybrid-rnn-mem-rwkv-fable5-gpt5.5-v1
.venv/bin/python integrations/delta_mem_rwkv_ms/gguf/materialize_memory_gguf.py --force
.venv/bin/python integrations/delta_mem_rwkv_ms/gguf/compare_memory_checkpoints.py

To generate and validate the isolated RWKV-MS math fixture from the sidecar-rebuilt checkpoint:

.venv/bin/python integrations/delta_mem_rwkv_ms/gguf/generate_rwkv_ms_math_fixture.py \
  --output .openresearch/artifacts/rwkv_ms_math_fixture.json
.venv/bin/python integrations/delta_mem_rwkv_ms/gguf/validate_rwkv_ms_math_fixture.py \
  --fixture .openresearch/artifacts/rwkv_ms_math_fixture.json \
  --json

The current fixture uses real layer-0 adapter tensors, covers projection, read-before-write state update, readout, and active q,o delta heads, and validates with max_abs_diff: 0.0. It is a PyTorch golden math fixture for a future GGML port, not stock llama.cpp memory execution.

The local llama.cpp checkout has an isolated C++ fixture for the compact sidecar:

/run/media/xiaol/B214449214445C0B/tools/cmake-4.3.3/bin/cmake \
  --build /run/media/xiaol/B214449214445C0B/tools/llama.cpp/build-cuda \
  --target test-rwkv-ms-fixture -j 8

/run/media/xiaol/B214449214445C0B/tools/llama.cpp/build-cuda/bin/test-rwkv-ms-fixture \
  .openresearch/artifacts/rwkv_ms_math_fixture.json \
  /run/media/xiaol/B214449214445C0B/models/gguf/gemma-4-E4B-it/gemma-4-E4B-it-rwkv-ms-memory.gguf \
  1e-5 1e-5

Current strict sidecar result: {"ok":true,"compared":51,"sidecar":true,"max_abs_diff":1.37090683e-06}. The no-sidecar run also passes with compared=11 and max_abs_diff=5.96046448e-08. This covers tests/test-rwkv-ms-fixture.cpp in llama.cpp parsing the compact sidecar, computing memory projections, HRMRWKV7LowRankCore feature projections, driving a second C++ read-before-write scan from those sidecar/GGML tensors, graph readout from the scan raw_reads plus graph feature_g, and delta_q/delta_o from the graph-produced readout. This fixture remains the isolated math parity check; the separate llama-completion smoke above is the Gemma4 generation runtime check.

The local llama.cpp checkout also has tests/test-rwkv-ms-state.cpp for the RWKV-MS recurrent state payload. It checks v2 state metadata, deterministic sidecar fingerprint validation, staged sidecar-local restore, and rejection for metadata/fingerprint/length mismatches. The fingerprint now includes the bound base GGUF hash, so slot files created before that binding should be regenerated. Full and sequence state restore now snapshot the current context before RWKV-MS-enabled loads and roll back that snapshot if the normal memory portion loads but the RWKV-MS sub-state fails. Failed server slot restore still clears the affected slot/context state after the library rollback and returns the exact state-load error. Context-owned memory mutation now uses llama.cpp llama_context_memory_* wrappers in the patched paths: clear and supported full-sequence removal keep RWKV-MS state synchronized, while unsupported sequence copy, keep, shift, and division fail explicitly under RWKV-MS instead of mutating only KV cache.

To generate the first PyTorch golden trace from the sidecar-rebuilt checkpoint:

.venv/bin/python \
  integrations/delta_mem_rwkv_ms/gguf/generate_reference_trace.py \
  --max-new-tokens 64 \
  --output .openresearch/artifacts/gguf_reference_trace_from_sidecar_64.json \
  --save-snapshot-dir .openresearch/artifacts/gguf_reference_snapshot_from_sidecar_64

To compare the running GGUF backend against that reference trace:

LLAMA_RWKV_MS=1 \
LLAMA_MODEL=gemma-4-e4b-it-rwkv-ms-q8 \
.venv-ui/bin/python tools/compare_gguf_to_reference_trace.py \
  --output .openresearch/artifacts/gguf_ui/trace_compare_reasoning_off.jsonl

With LLAMA_REASONING=off, the comparison harness can log either base-GGUF or RWKV-MS-sidecar runs. Stock llama.cpp still does not execute RWKV-MS memory; the sidecar mode requires the local patched branch.

Acknowledgement

This work builds on the Log-Linear Attention repository and uses local HRM-Text/RWKV memory ideas as mechanism baselines. The added experiments are intended for controlled research exploration, not as a trained-model benchmark.