loracle-pretrain-v7-sweep-A-step1875

loracle-pretrain-v7-sweep-A-step1875

Step-1875 (30% of epoch 1) checkpoint from v7 sweep A training run. Best checkpoint on disk at the time the run crashed post-eval-cycle.

Training config

• Base: Qwen3-14B (frozen)
• Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
• Direction tokens: svd_fixed_k16_mag7_rankfirst, 4480 tokens per LoRA
• Prefix mode: rank_tagged
• Data: ceselder/loracle-pretrain-mix (25k orgs, ~2 QA rows each = 50k train rows, 300 orgs for eval)
• Effective batch = 8 (batch_size=1 x grad_accum_steps=8)
• LR = 3e-5, linear schedule, warmup = 500 opt-steps (8.9% of training)
• Epochs = 1 (target 6,250 opt-steps total; step-1875 = 30% mark)

Eval numbers at step 1875 (canonical IA-paper judge rubric, bugfixed parser)

Judge: Sonnet 4.6 via OpenRouter.

Trajectory: 20.8% (step 625) -> 21.8% (step 1250) -> 29.1% (step 1875). Climbing strongly. Training stalled in a post-eval-cycle Python GC/thread-leak apparent-hang that required SIGKILL; weights preserved from the "new best" save.

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/vhwb7yvr

Layout

• interpreter/ PEFT LoRA adapter (load with PeftModel.from_pretrained)
• encoder.pt AO encoder state_dict
• ao.pt AO norm-match hook params
• tokenizer/ Qwen3-14B tokenizer
• loracle_config.yaml Training config snapshot