loracle-pretrain-v7-sweep-A-step625

loracle-pretrain-v7-sweep-A-step625

Step-625 (10% of epoch 1) checkpoint from v7 sweep A training run.

Training config

• Base: Qwen3-14B (frozen)
• Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
• Direction tokens: svd_fixed_k16_mag7_rankfirst, 4480 tokens per LoRA
• Prefix mode: rank_tagged
• Data: ceselder/loracle-pretrain-mix (25k orgs, ~2 QA rows each = 50k train rows, 300 orgs for eval)
• Effective batch = 8 (batch_size=1 x grad_accum_steps=8)
• LR = 3e-5, linear schedule, warmup = 500 opt-steps (8.9% of training)
• Epochs = 1 (target 6,250 opt-steps total; step-625 = 10% mark)

Eval numbers at step 625

Judge: Sonnet 4.6 via OpenRouter.

Caveat: auditbench 0% was under an overly strict custom rubric that has since been reverted to the canonical IA-paper rubric. Rejudging the saved rollouts under the standard rubric should yield a much higher AB number (v6_A at a comparable stage was 60%+ under the paper rubric).

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/vhwb7yvr

Layout

• interpreter/ PEFT LoRA adapter (load with PeftModel.from_pretrained)
• encoder.pt AO encoder state_dict
• ao.pt AO norm-match hook params
• tokenizer/ Qwen3-14B tokenizer
• loracle_config.yaml Training config snapshot