Downloads · 30 days
0
loracle-pretrain-v7-sweep-A-oneq-final-step3120
ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120
loracle-pretrain-v7-sweep-A-oneq-final-step3120 is a machine learning model from ceselder. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
End-of-epoch (final) checkpoint from v7 sweep A "oneq" variant (1 randomly-selected QA per organism, half the data of v7A).
Access
Public
Updated Apr 25, 2026
Repo size
2.1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.1 GB · 99%
At a glance
- Access
- Public
- Created
- Apr 25, 2026
- Updated
- Apr 25, 2026
- SHA
- f8ae209a
Model card
From the Hugging Face model README
loracle-pretrain-v7-sweep-A-oneq-final-step3120
End-of-epoch (final) checkpoint from v7 sweep A "oneq" variant (1 randomly-selected QA per organism, half the data of v7_A).
Training config
- Base: Qwen3-14B (frozen)
- Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
- Direction tokens: svd_fixed_k16_mag7_rankfirst, 4480 tokens per LoRA
- Prefix mode: rank_tagged
- Data: ceselder/loracle-pretrain-mix subsampled to 1 random QA row per organism (seed=42 deterministic shuffle+drop_duplicates) → 25k train rows
- Eval: 300 holdout orgs (1 row each)
- Effective batch = 8 (batch_size=1 x grad_accum_steps=8)
- LR = 3e-5, linear schedule, warmup = 500 opt-steps
- Epochs = 1, total 3125 opt-steps; this checkpoint at step 3120 = last eval cycle (100%)
Eval numbers at step 3120 (final)
Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric.
| Set | organisms | any-match |
|---|---|---|
| heldout_ia | 20 | 40.0% |
| trigger_recovery_heldout_ia | 20 | 15.0% |
| auditbench | 56 | 30.4% |
| ood_models_v3 | 27 | 40.7% |
| val/mean_all_evals | - | 31.5% |
train/loss: 0.62 (low — well-converged) val/loss: 1.95
Full trajectory of the oneq run
| step (% epoch) | examples_seen | mean | comments |
|---|---|---|---|
| 312 (10%) | 2,496 | 16.3% | first eval |
| 624 (20%) | 4,992 | 27.9% | |
| 936 (30%) | 7,488 | 26.1% | |
| 1248 (40%) | 9,984 | 30.0% | |
| 1560 (50%) | 12,480 | 32.7% | best (AB peak 41.1%) |
| 1872 (60%) | 14,976 | 23.5% | dip |
| 2184 (70%) | 17,472 | 28.2% | |
| 2496 (80%) | 19,968 | (filled in by run) | |
| 2808 (90%) | 22,464 | (filled in by run) | |
| 3120 (100%, final) | 24,960 | 31.5% | end of epoch |
Comparison vs v7_A (2 QA/org, 50k rows)
v7_A best (step-5625): mean 35.1% (heldout 45, trigger 30, AB 32.1, ood 33.3) oneq final (step-3120): mean 31.5% (heldout 40, trigger 15, AB 30.4, ood 40.7)
oneq has notably stronger ood_models_v3 (40.7% vs 33.3%) but weaker heldout_ia + trigger_recovery.
Wandb
Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/wixyr3fi
Layout
- interpreter/ PEFT LoRA adapter
- encoder.pt AO encoder state_dict
- ao.pt AO norm-match hook params
- tokenizer/ Qwen3-14B tokenizer
- loracle_config.yaml Training config snapshot
- Created
- Apr 25, 2026
- Updated
- Apr 25, 2026