hnet-chunking-spec-82m

H-Net dynamic-chunking spec models (82.5M) — with full training trajectories

Thirty-one byte-level H-Net runs at 82,568,832 parameters, one hierarchy stage, trained to study what a learned chunker discovers when it is free to choose its own computational granularity — and, unlike the 22.5M pilots, when it discovers it.

The pilots at AdaptiveChunking/hnet-chunking-pilots retained only a final checkpoint. These runs retain the whole log-spaced trajectory, which is the point: it is what makes training-dynamics and activation-patching questions answerable at multiple steps rather than at one. That trajectory cannot be reconstructed after the fact.

Runs

The exp32_* runs are controls (experiments/exp32_train_controls.py monkey-patches the forward; weights load into the same HNet class but must be run with the same patch to reproduce their BPB — see results/exp32_trained_controls/RESULTS.md).

Data, seed schedule and FLOP budget are held identical within a scale; only the chunker's objective varies. Architecture: d_enc 384, d_main 768, n_main 10, seq_len 1024, batch 64 (specfull), corpus FineWeb2 + FineWeb-v1 (en).

What they show

The parity effect replicates at 3.7× the pilot's parameters, and at the full data budget, with essentially no likelihood cost:

At 0.72 GB: same direction, same magnitude, same chunks-per-sentence vs chunks-per-character asymmetry as the pilot, and condition C's lock-in reversion (+0.115) replicates to the decimal. At 5 GB the freeze does not bank the effect: the router's W_q/W_k are frozen at step 53,410 but the encoder that feeds them is still trained, and in two of three seeds the allocation drifts back toward the unregularised one over the 22,890 frozen steps (end-of-phase-2 Gini 0.066–0.103 in all three; end-of-run 0.104 / 0.181 / 0.281). The equalisation is a property of the objective being on, not a state that can be frozen in. Mean BPB drops from 1.376 (0.72 GB) to 1.153 (5.00 GB), so the full-budget models are genuinely better trained, not just longer.

Format

Each run directory holds {step:06d}.safetensors (one per retained checkpoint), config.json, final.json and log.jsonl. config.json carries model_config, the training config, a checkpoints list mapping tag → step, and a tied_weights map. emb.weight and head.weight are tied, so only emb.weight is stored — restore with:

import json
from safetensors.torch import load_file

sd = load_file("spec_A_s0/011000.safetensors")
cfg = json.load(open("spec_A_s0/config.json"))
for dst, src in cfg["tied_weights"].items():
    sd[dst] = sd[src]

manifest.json at the repo root lists every run with its parameter count, condition, seed, final step and checkpoint count.

Probe trajectories for these runs (253 npz per 11k-step run, 361 / 529 per full run) live alongside the pilots' in AdaptiveChunking/hnet-chunking-probes under the same spec_* / specfull_* names.

Known issues — carried over from the pilots, and still true here

These are documented rather than hidden, because they change what the logged fields mean:

• mask and mask_level1 are byte-for-byte identical. Everything here is a single-stage model (S=1); mask_level1 is a placeholder, not a second hierarchy level. Any analysis treating it as one is measuring the same thing twice.
• Thai len(mask) runs 21 bytes long relative to the text it indexes.
• Boundaries phase-lock sub-character on several scripts — the router does not respect codepoint edges, which is expected for a byte-level model but matters when aligning chunks to linguistic units.
• Analyses of these runs so far find that capability crystallises before segmentation does, that learned chunks are not functionally localised, and that the units are not linguistic. Treat "chunk" as "a unit of compute allocation", not "a morpheme".

Caveats

• Conditions C, D, E exist only at the 0.72 GB budget; at 5 GB only A and B were run (3 seeds each), plus the exp32 one-chunk control (1 seed).
• freeze_scope: router+encoder — the arm the 5 GB freeze failure calls for — has not been run.
• Reported Gini is over chunks per content-matched sentence, computed on 12 languages with high-resource = en, de, ru, zh. A Gini is not comparable across different language sets.

Provenance

Produced by tier1/train.py in beetle-hnet; launched by tier1/launch_spec_overnight.sh, launch_spec_cde.sh, launch_spec_full.sh, launch_spec_full_seeds.sh and (exp32) scratch/logs/r4/lane_exp32.sh; packaged by tier1/push_spec_to_hf.py. Analysis in results/exp21_tier1_spec/ and results/exp32_trained_controls/.