Downloads · 30 days
94
55% of all-time downloads
yuyu0529nya/qwen2.5-7b-tau2-airline-sft-lora
qwen2.5-7b-tau2-airline-sft-lora is a text generation model from yuyu0529nya. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
The RL warm-start checkpoint for multi-turn tool-agent RL on τ²-bench's airline domain.
Downloads · 30 days
94
55% of all-time downloads
All-time downloads
172
Public
Repo size
162 MB
Likes
0
Public
Click a slice to open those files.
.safetensors162 MB · 100%
From the Hugging Face model README
The RL warm-start checkpoint for multi-turn tool-agent RL on
τ²-bench's airline domain.
Behavior-cloned from DeepSeek V4 Flash teacher trajectories, collected by running the teacher as the agent through the exact same τ²-bench harness used for training and evaluation.
The collection script requests
deepseek-chat, which is DeepSeek's alias for their current chat model. Every raw API response in the training data reportsmodel: deepseek-v4-flash(1156/1156 assistant turns, 671/671 user-simulator turns), so the teacher is V4 Flash, not V3. Stated here from the data rather than from the script.
GRPO on τ²-bench-airline from raw Qwen2.5-7B-Instruct is null — and not because of a bug.
Base success is ~20%, so most rollout groups come back all-fail → zero intra-group variance → the
group-relative advantage is ~0 → no gradient. RL sharpens what a policy already does sometimes;
it cannot invent a skill. Worse, the base policy's own successful rollouts never invoke the
write-tools (update_reservation_*) that held-out tasks require, so self-sampling can't bootstrap
them either.
This adapter is the floor that makes the RL step work.
This adapter, merged into the base, is the step 0 policy of the
tau2_airline verl recipe. Measured under
that recipe's protocol (held-out BINARY mean@4, 20 held-out airline tasks, VAL_TEMP=0.5,
llama-3.3-70b user simulator):
| val@0 (this adapter) | val@20 (after GRPO) | |
|---|---|---|
| seed 42 | 0.275 | 0.5625 |
| seed 123 | 0.375 | 0.55 |
GRPO endpoint across 2 seeds: 0.556 ± 0.01. Measured evaluation noise on this same checkpoint (6 repeats of val@0) is 0.30 ± 0.05, so the two val@0 values above are a single checkpoint sampled twice from that band, not a difference.
Scope note. A separate hand-written pipeline reported
base 0.20 → distilled SFT 0.405 → GRPO 0.545for this approach, but under a different eval protocol (pass^1 over 10 trials with a self-hosted 7B user simulator, n=200). Those numbers are not directly comparable to the table above — different user simulator, different metric — and I have not re-verified that this exact file is the checkpoint they were measured on. The table above is what this file scores.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "yuyu0529nya/qwen2.5-7b-tau2-airline-sft-lora")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
To use it as an RL starting point, merge first (model.merge_and_unload()) and point the trainer at
the merged weights — that is exactly how the recipe above consumes it.
This adapter only makes sense inside the τ²-bench airline harness — same system prompt, same 14 tool schemas, same dialogue loop. It is a policy for that environment, not a general chat model.
| base | Qwen/Qwen2.5-7B-Instruct |
| method | behavior-cloning SFT (QLoRA 4-bit) on teacher trajectories |
| teacher | DeepSeek V4 Flash (deepseek-v4-flash, requested via the deepseek-chat alias) — agent and user-simulator, temp 0.7 |
| data | yuyu0529nya/tau2-airline-deepseek-distill (ds_114 config) — 114 successful teacher trajectories (reward = 1), deduped by action signature, capped 4/task |
| LoRA | r=16, alpha=32, dropout=0.05, targets = q,k,v,o,gate,up,down_proj |
| loss | assistant-token-only (render-twice-diff mask) |
Teacher success on these tasks was ~76% vs base ~20%, and 51% of teacher trajectories invoke the write-tools the base policy never reaches on its own. That skill gap is why self-RFT from base cannot work here, and why distillation — rather than more RL — was the fix.
Qwen2.5-7B-Instruct, Apache-2.0 → derivatives unrestricted.@example.com, RFC 2606 reserved). No real PII.llama-3.3-70b. Not controlled for.airline domain only. Not evaluated on other τ²-bench domains. Not a general-purpose model.