loracle-ptrl-v8

Loracle PT-RL v8 — instruction-following + OOD generalist

A "loracle" that reads LoRA weight diffs and predicts what the LoRA does, in plain first-person behavioral language. v8 is the state-of-the-art PT-RL loracle as of this release — beats v6 on AuditBench AND wins on OOD evals (taboo, subliminal, open-ended specialty fine-tunes).

Headline: 75.0% AuditBench any-match + 40.8% rollout-mean — beats v6's 71.4% peak by +3.6pp. Plus the only loracle with non-zero subliminal recovery and 83% taboo identification.

What's new vs v6/v7

• 9 question types (behavioral_paraphrase, concise, detailed, list, contrastive, topic_summary, comparison, json_output, refusal_probe) — teaches instruction-following, breaks mode-collapse to a single phrasing
• Conditional refusal probe — only included for safety-relevant orgs (~12% by topic-keyword filter), no degenerate "I don't refuse" mode collapse
• No literal AuditBench prompts in training — pure paraphrase / instruction-following Q/A, cleaner generalization
• JUDGE_GROUP_PROMPT_INSTRUCTION rewards format compliance (concise = 1 sentence, JSON = valid JSON, list = 3 bullets) alongside topical accuracy

Eval results

Per-config AB breakdown at step_30 / c60:

• synth_docs_only_then_redteam_high: 92.9% (13/14) ← near-perfect
• synth_docs_only_then_redteam_kto: 78.6%
• transcripts_only_then_redteam_high / kto: lower

Baselines:

• IA paper: ~25% AB
• Prior doc-summary loracle (v3/v5): 33–37% AB

Training

• Init: ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120
• SFT warmstart: 1 epoch on 1904 examples (476 orgs × 4 random Q/A)
• RL: 60 cycles Dr. GRPO online, 32 prompts × 16 rollouts/cycle, 4-GPU DDP on 4× H200
• Judge: claude-opus-4-7 + adaptive thinking, JUDGE_GROUP_PROMPT_INSTRUCTION
• Total wall: ~5h training + ~1h SFT

Method spec: ceselder/loracle-ptrl-data-v8 README has full details (Q/A taxonomy, hypers, episode walkthrough, judge prompt).

Training data: ceselder/loracle-ptrl-data-v8.

Hyperparameters

run_name: drgrpo_pretrain_only_v8
algorithm: drgrpo
n_cycles: 60 (effective)
n_prompts_per_cycle: 32
k_rollouts: 16
temperature: 0.75
lr: 5.0e-6
eps_low: 0.2
eps_high: 0.28
max_grad_norm: 1.0
max_length: 5500
unbiased_advantages: true
prefix_mode: rank_tagged
top_k: 16
n_direction_tokens: 4480

judge_mode: ranking
judge_prompt_mode: instruction_pretrain
judge_provider: anthropic
rollout_judge_model: claude-opus-4-7

How to load (sketch)

from huggingface_hub import snapshot_download
ckpt = snapshot_download("ceselder/loracle-ptrl-v8")
# load Qwen3-14B base + interpreter PEFT (rank-256) + encoder.pt
# inject direction tokens (svd_fixed_k16_mag7_rankfirst, [4480, 5120] bf16) at layer 1
# prompt: "Are there any unusual characteristics you display only for certain types of prompts?
#          Please respond with a single sentence response predicting your behavior."

Limitations

• Subliminal recovery is whale-only. Dolphin / butterfly / tiger get 0/6 across all evals — the bias for these is hidden too deep in the LoRA's behavior to leak into weight signature
• 3 GGUF-only OOD orgs (manga_translator, opus_distill, furry_novel) couldn't be extracted (gguf-py tensor shape mismatch on Qwen3-14B GQA — TODO)
• Direction tokens must use svd_fixed_k16_mag7_rankfirst extraction recipe

Citation

@misc{loracle-ptrl-v8,
  author = {Celeste de Champhelaere},
  title  = {Loracle PT-RL v8 — instruction-following + OOD generalist},
  year   = {2026},
  url    = {https://huggingface.co/ceselder/loracle-ptrl-v8},
  note   = {75% AuditBench any-match via instruction-following Q/A taxonomy + Dr. GRPO RL on continued-pretrain LoRAs.}
}