rlm-explorer-qwen3-8b-qasper-sft

Qwen3-8B QASPER code-agent SFT checkpoints

This private repository preserves two experimental QLoRA checkpoints from the RLM Explorer scientific-paper research-agent project. The model was trained to write Python actions that call search(), read(), and extract() before submitting an evidence-grounded answer.

Research status: these checkpoints are artifacts for evaluation and debugging. They do not currently demonstrate an improvement over base Qwen3-8B. Epoch 2 improved held-out next-token metrics but collapsed to immediate answer submission in behavioral evaluation.

Repository contents

Each checkpoint contains the LoRA adapter, tokenizer, trainer state, optimizer, scheduler, RNG state, and training arguments. The adapter in the original final/ directory was byte-identical to checkpoint-96, so only one epoch-2 copy is stored here.

Training

• Base model: Qwen/Qwen3-8B
• Data: 189 training and 47 validation multi-turn QASPER trajectories
• Objective: supervised fine-tuning with loss on assistant tokens
• Quantization: 4-bit NF4 with double quantization and bf16 computation
• LoRA: rank 4, alpha 8, dropout 0.05
• Targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
• Learning rate: 2e-4
• Effective batch: batch size 1 with 4 gradient-accumulation steps
• Maximum sequence length: 8192 tokens
• Hardware: one NVIDIA L4

Token-level validation

These metrics show that the model learned to imitate held-out trajectories. They do not establish that it learned to operate the research-agent loop.

Behavioral evaluation

The frozen evaluation contained 40 QASPER test questions: 20 answerable from the paper corpus and 20 requiring abstention. There was no training/test question or paper overlap. Both policies used deterministic decoding, seed 42, a 10-step limit, and the same 416-paper corpus.

The preregistered abstention threshold required at least 7/20 correct abstentions and no more than 4/20 false abstentions. Epoch 2 did not pass (p = 1.0, two-sided Fisher exact test).

The first full comparison contained a train/serve mismatch: the initial observation repeated the tool preamble. A corrected five-question diagnostic removed that duplication. Epoch 2 still submitted immediately on 5/5 questions and used no tools, although it no longer copied placeholder text. This isolates the central failure: the adapter learned a direct-submit shortcut.

Loading an adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3-8B"
repo_id = "jasonlingg/rlm-explorer-qwen3-8b-qasper-sft"

tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder="epoch-1")
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, repo_id, subfolder="epoch-1")

Use epoch-1 or epoch-2 explicitly. Do not treat epoch-2 as a recommended production model.

Reproducibility

• Training/evaluation project: rlm-explorer
• Full base/epoch-2 evaluation code commit: d7e6efa2b9f90af32584f6973eff364389cbf8ae
• Aligned diagnostic code commit: af762ba0ab63e179f4c0ba9714e0c4a7d9985793
• Base-model revision: b968826d9c46dd6066d109eabc6255188de91218
• Evaluation question SHA-256: 5720a5220ff9e47eb2465028d9592551d4575830ec476dfafc31a4470a11d4d0
• Evaluation corpus SHA-256: ee9e3b2986845194cf9a25dbd1ea1ab2cb8e44cd5e385e5cefcb8f7665335a9f

The files under evaluation/ contain the exact question IDs, policy settings, hardware, software versions, run IDs, and full trajectories.