Downloads · 30 days
0
loracle-pretrain-v7-sweep-A-oneq-step1560
ceselder/loracle-pretrain-v7-sweep-A-oneq-step1560
loracle-pretrain-v7-sweep-A-oneq-step1560 is a machine learning model from ceselder. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Step-1560 (50% of epoch 1) checkpoint from v7 sweep A "oneq" variant (1 randomly-selected QA per organism, half the data of v7A). Best-so-far in this run when uploaded; AuditBench peaked at 41.1% here.
Access
Public
Updated Apr 25, 2026
Repo size
2.1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.1 GB · 99%
At a glance
- Access
- Public
- Created
- Apr 25, 2026
- Updated
- Apr 25, 2026
- SHA
- e27103be
Model card
From the Hugging Face model README
loracle-pretrain-v7-sweep-A-oneq-step1560
Step-1560 (50% of epoch 1) checkpoint from v7 sweep A "oneq" variant (1 randomly-selected QA per organism, half the data of v7_A). Best-so-far in this run when uploaded; AuditBench peaked at 41.1% here.
Training config
- Base: Qwen3-14B (frozen)
- Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
- Direction tokens: svd_fixed_k16_mag7_rankfirst, 4480 tokens per LoRA
- Prefix mode: rank_tagged
- Data: ceselder/loracle-pretrain-mix subsampled to 1 random QA row per organism (seed=42 deterministic shuffle+drop_duplicates) → 25k train rows
- Eval: 300 holdout orgs (1 row each)
- Effective batch = 8 (batch_size=1 x grad_accum_steps=8)
- LR = 3e-5, linear schedule, warmup = 500 opt-steps
- Epochs = 1, total 3125 opt-steps; this checkpoint at step 1560 = 50% mark
- examples_seen at this checkpoint: 1560 x 8 = 12,480
Eval numbers at step 1560
Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric.
| Set | organisms | any-match |
|---|---|---|
| heldout_ia | 20 | 40.0% |
| trigger_recovery_heldout_ia | 20 | 20.0% |
| auditbench | 56 | 41.1% |
| ood_models_v3 | 27 | 29.6% |
| val/mean_all_evals | - | 32.7% |
Trajectory of the oneq run so far
| step (% epoch) | examples_seen | mean | comments |
|---|---|---|---|
| 312 (10%) | 2,496 | 16.3% | first eval |
| 624 (20%) | 4,992 | 27.9% | |
| 936 (30%) | 7,488 | 26.1% | |
| 1248 (40%) | 9,984 | 30.0% | |
| 1560 (50%) | 12,480 | 32.7% | AB peak 41.1% |
| 1872 (60%) | 14,976 | 23.5% | dip |
| 2184 (70%) | 17,472 | 28.2% |
Wandb
Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/wixyr3fi
Layout
- interpreter/ PEFT LoRA adapter
- encoder.pt AO encoder state_dict
- ao.pt AO norm-match hook params
- tokenizer/ Qwen3-14B tokenizer
- loracle_config.yaml Training config snapshot
- Created
- Apr 25, 2026
- Updated
- Apr 25, 2026