loracle-pretrain-v4-sweep-A-step2598

Loracle — pretrain v4 sweep A, full epoch (step 2598)

A loracle is a model that reads another fine-tuned model's LoRA weight deltas (as direction tokens projected into the residual stream) and describes the behaviors encoded in those weights, without running the target model.

This checkpoint is the full-epoch (step 2598 = 1.0 epoch) snapshot of loracle_pretrain_sft_v4_sweep_A.

• Base model: Qwen/Qwen3-14B
• Interpreter adapter: rank-256 PEFT LoRA with rslora=True, α=32
• Direction-token input: 4480-token prefix in (rank, layer, mag7) rank-first layout, produced by extract_svd_fixed_tokens from any Qwen3-14B LoRA / full-FT / GGUF (via extract_hf/download_and_extract_hf_models.py).
• Training corpus: 2500 synthetic pretrain organisms (fineweb + redpajama2 + misc) from ceselder/loracle-pretrain-qa-v4.1-25k, one QA pair per organism × 2 = ~5000 training examples.
• Hook: layer-1 AO norm-match injection (h' = h + ‖h‖ · v̂).
• Prefix mode: rank_tagged (16 <SVD N:> markers × 280 placeholder slots per rank).

wandb run

https://wandb.ai/adamkarvonen/lora-oracles/runs/20tqy8is

Eval numbers (at step 2598)

Note: these numbers were collected with BUGGY (pre-reshape-fix) eval tokens. Re-eval against corrected tokens via src/train_loracle/eval_ckpt.py.

• val/loss: 1.907
• val/mean_all_evals: 32.3%
• auditbench rollout_mean: 19.9%
• heldout_ia rollout_mean: 21.1%
• ood_models_v3 rollout_mean: 14.4%
• trigger_recovery: 0% (keyword-match judge was too strict; use semantic rubric in newer eval yamls)

Repo contents

• interpreter/ — PEFT adapter (load with PeftModel.from_pretrained(base, "interpreter"))
• tokenizer/ — HF tokenizer (inherits Qwen3-14B's, no vocab extension)
• encoder.pt — AOEncoder state-dict (no learnable params; kept for API symmetry)
• ao.pt — training metadata (step, val_loss, val_mean_all_evals)
• loracle_config.yaml — full training config snapshot

Loading

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "ceselder/loracle-pretrain-v4-sweep-A-step2598/interpreter")
tokenizer = AutoTokenizer.from_pretrained("ceselder/loracle-pretrain-v4-sweep-A-step2598/tokenizer")

See https://github.com/ceselder/loracles for the full inference + eval pipeline.