noesis-rwkv7-g1h-2.9b

noesis — RWKV-7 G1h 2.9B, state-regularized (A1 pilot)

Fine-tuned from BlinkDL/rwkv-7-g1h (2.9B) using trajectory-regularized state training — a novel RWKV-specific training objective that shapes the WKV state trajectory to carry richer information per token, particularly inside <think> reasoning spans.

Research project: github.com/Vaniell0/noesis. Goal: understand how an RNN state machine can learn to reason rather than retrieve — exploiting RWKV-7's recurrent state as a working-memory substrate.

Checkpoints

Best documented result (step9 epoch 0, not in this upload — checkpoint lost to disk overflow during training): 43.75% overall, extraction 62.5%, symbolic 75%, arithmetic 75%. Eval JSON is included in the repo under experiments/A0_eval/results/step9_e0_eval_np512.json.

Recommendation: use e0 for continued fine-tuning (better extraction/reasoning balance). Use e1 if you specifically need symbolic or arithmetic performance.

Baselines (A0 eval, 48 tasks, np=512)

Note: G1h base scores 39.6% at np=512 — already competitive. The fine-tune shifts the profile (symbolic 25%→75%, scheduling 16.7%→50%) but trades bit_decoding (18.8%→0%). The bit_decoding regression is a corpus artifact: DSL-format action-chain training suppresses the multi-step bitwise CoT the base model uses.

Method

L_state — trajectory regularization

WKV state penalty that rewards state motion rather than suppressing it — sign-flipped slow-feature analysis applied to per-layer states. For each token t ≥ 2:

L_state(t) = Σ_L w_L × (
    −λ_δ · ‖s_L(t) − s_L(t−1)‖₂          # delta: reward state change
    −λ_κ · ‖(s_L(t)−s_L(t−1)) − (s_L(t−1)−s_L(t−2))‖₂  # curvature
)

Layer weights from A0.5 causal probing (KL under zero-layer corruption): w_4=0.035, w_16=0.506, w_20=0.238 (L16 dominates — consistent across scales).

ε-mask

L_state applied with different weights per token span:

• Inside <think>…</think>: full weight (α=1e-3)
• Outside <think>: 5% weight (α=5e-5)

Effect: gradient strongly shapes WKV state motion during reasoning spans; answer tokens receive weak signal and remain clean for extraction.

H12b.i aux loss

loss = −mean_entropy(per-rank importance scores across LoRA pairs)

Prevents LoRA rank collapse — keeps all 32 rank slots active. Importance score per rank = ‖lora_A[rank, :]‖ × ‖lora_B[:, rank]‖.

Key experimental findings

N-pass re-feed sweep (H10): Silent re-feeds before final decode (no output, just state accumulation):

• N=1: 27.1% — single pass baseline
• N=2: 33.3% — consistent improvement across checkpoints
• N=3: 6.3% — catastrophic collapse (−27pp)

N=3 collapse mechanism: DSL-format training creates an implicit <tool_call> attractor. By the third pass, WKV state has accumulated a "waiting for tool_result" pattern that overwrites the reasoning signal. The model has no self-termination signal.

Single bit_decoding solve: At N=2, K=128, state_readout mode — one bit_decoding task (bit_sub_01) was solved by reasoning through bit positions across two passes. Zero solved in single-pass mode. This is the only bit_decoding success in the full sweep, suggesting the problem is compute-depth rather than knowledge.

H8/H9 — state computation confirmed (causal probe): 1152 paired forwards, 4 corruption types. G1h 2.9B vs RWKV-7 World 2.9B:

• σ-slopes: G1h 1.58–1.67 (superlinear) vs World 1.13–1.19
• Cross-prompt/noise ratio: 34–40× (state carries prompt-conditional structure)
• G1h KL at σ=0.1: 40× higher than World
• Layer hotspots: L4, L16, L20 — consistent across 0.4B and 2.9B

H12b — multi-slot working memory (base capability): G1h 2.9B base holds K=8 parallel slots at 53% accuracy (P=1 retrieval) without any working-memory training. G1d 0.4B base: ~11%. The 5× gap is an emergent property of the G1 pre-training recipe. Action-chain fine-tuning (step7) hurt this: 7/9 cells regressed (K=2,P=1: base 65% → step7 25%). Specialisation to one-shot slot access costs depth retention.

Training details

Step9 corpus: 297 RFC binary-protocol QA tasks (TCP, SCTP, NTP, HTTP/2, OSPF, BGP, …), 268 train rollouts, 71k tokens, plain CoT format with <think> spans and state_mask tensor. ctx_len=32768, LoRA rank=32 α=64, 1 epoch on RTX 4090 (~5 hours).

Step9b corpus: 6-source mix — RFC QA (25%), HH-RLHF (20%), ReAct (15%), action chains (15%), self-generated CoT (15%), base language (10%). 836 rollouts, 226k tokens. ctx_len=512 — this was a mistake: at ctx_len=512, the trajectory window has T<3 chunks and L_state collapses to near-zero gradient. Extraction regressed (62.5%→50% e0, 12.5% e1) as a result.

Usage

from rwkv.model import RWKV
from rwkv.utils import PIPELINE, PIPELINE_ARGS

# Note: pass path WITHOUT .pth extension — rwkv package appends it
model = RWKV(model="path/to/rwkv-step9b-e0", strategy="cuda bf16")
pipeline = PIPELINE(model, "rwkv_vocab_v20230424")

prompt = """<|im_start|>system
You are a precise protocol analyst. Work step by step.<|im_end|>
<|im_start|>user
RFC 793 TCP header:
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |          Source Port          |       Destination Port        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                        Sequence Number                        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

What is the bit offset and width of Destination Port?<|im_end|>
<|im_start|>assistant
<think>"""

# RWKV_CUDA_ON=0 if cuBLAS headers unavailable (use PyTorch WKV ops)
import os; os.environ["RWKV_CUDA_ON"] = "0"

args = PIPELINE_ARGS(temperature=0.1, top_p=0.9, token_count=512)
output = pipeline.generate(prompt, token_count=512, args=args)
print(output)

Hypotheses

Active hypotheses with data:

• H8 (state-as-computation): CONFIRMED — causal probe shows G1h state does structured work, not rolling summary
• H9 (G1-line amplifies state): CONFIRMED — G1h 40× higher KL response than World at same scale
• H10 (N×K×mode frontier): PARTIAL — N=2 sweet spot, N=3 always collapses; readout axis degenerate without dedicated training. RWKV exposes 3 orthogonal test-time compute dials (N, K, mode) that transformers cannot — see docs/effort-frontier.md
• H12b (multi-slot working memory): PARTIAL — base capacity exists (53% at K=8), fine-tuning degrades it without explicit multi-slot supervision
• H16 (emit gate): UNTESTED — N=3 collapse argues this is load-bearing; gating signal needed before N>2 is usable
• H20 (aporia/contradiction holding): PARTIAL — p(neither)=0.746 on 100 contradictory prompts (G1d 0.4B); modal collapse rate 0.54
• H21 (premise validity readout): PARTIAL — G1h 2.9B test F1=1.00 (40-item, impossible+categorical invalids); G1d 0.4B F1=0.75
• H22 (attribution probe): PARTIAL — LOO F1=0.947 on 240-item set (G1d 0.4B); TP=115, FP=8, TN=112, FN=5

Full hypothesis tree (H1–H23): HYPOTHESES.md

Citation / contact

Research by Vaniello. Code: github.com/Vaniell0/noesis. Base model by BlinkDL / RWKV team.