loracle-pretrain-v7-sweep-A-step1250

loracle-pretrain-v7-sweep-A-step1250

Step-1250 (20% of epoch 1) checkpoint from v7 sweep A training run.

Training config

• Base: Qwen3-14B (frozen)
• Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
• Direction tokens: svd_fixed_k16_mag7_rankfirst, 4480 tokens per LoRA
• Prefix mode: rank_tagged
• Data: ceselder/loracle-pretrain-mix (25k orgs, ~2 QA rows each = 50k train rows, 300 orgs for eval)
• Effective batch = 8 (batch_size=1 x grad_accum_steps=8)
• LR = 3e-5, linear schedule, warmup = 500 opt-steps (8.9% of training)
• Epochs = 1 (target 6,250 opt-steps total; step-1250 = 20% mark)

Eval numbers at step 1250 (canonical IA-paper judge rubric, bugfixed parser)

Judge: Sonnet 4.6 via OpenRouter. Rubric: canonical JUDGE_PROMPT (same-category-of-behavior). Numbers below are from rejudge_eval_rollouts.py with the **Yes parser fix — a prior version counted "Yes..." responses as NO due to literal startswith("yes") missing the markdown bold prefix.

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/vhwb7yvr

Layout

• interpreter/ PEFT LoRA adapter (load with PeftModel.from_pretrained)
• encoder.pt AO encoder state_dict
• ao.pt AO norm-match hook params
• tokenizer/ Qwen3-14B tokenizer
• loracle_config.yaml Training config snapshot