ocular-qwen3-4b-mixed01

ocular-qwen3-4b-mixed01

DoRA finetunes of Qwen3-4B on a personal corpus, merged and quantized. Private. Two ranks trained identically except for adapter capacity — the pair is the point.

Training

base      Qwen/Qwen3-4B
method    DoRA over a 4-bit NF4 base (QLoRA-style), all 7 projections
seq       2048, 1 epoch, bf16, paged_adamw_8bit
data      22.08M tokens, 9 sources
hardware  RTX 5090, ~3.8h each

The rank comparison

Holdout is cut along whole tasks and whole languages, verified to have zero overlapping (task, language) pairs with training.

unseen-language items are tasks trained on in other languages; unseen-task items were never seen at all. The widening gap is the transfer signal.

Rank 32 captures 77% of it with a quarter of the parameters. The extra capacity shows up almost entirely in unseen-language — re-rendering known tasks. On genuinely novel tasks the two are tied (4.73 vs 4.82), and rank 128 fit the training set harder for that non-gain.

Known defects

1. Training records were concatenated with no separator, so block boundaries taught transitions between unrelated documents as if they were continuations. Conversations got partial cover from <|im_end|>; the ~6% that are plain documents got none.
2. Packing slices mid-record every 2048 tokens.
3. weight_decay = 0.0.
4. The 40% Rosetta share is probably too high. Asked "what is 17 times 23", the 1.7B sibling writes a C# program instead of answering 391.

Provenance

Trained 2026-08-21 on rented RTX 5090s; merged, converted and quantized on the same boxes. Trail in the source repo under runs/2026-08-21/.