ALF-qwen3B-privilege

ALF-Qwen3B Privileged Teacher

This repository contains a LoRA teacher adapter for Qwen/Qwen2.5-3B-Instruct, trained to select ALFWorld TextWorld actions from deployable state information X plus a structured privileged hint H.

The teacher is intended for privileged knowledge distillation research. It is not a deployable student: at inference time the teacher expects an oracle-derived, episode-level hint. The student in the accompanying experiment sees X only.

Model inputs and target

• X: task goal, recent interaction history, current observation, and admissible commands.
• H: a type-level task sketch containing the skill, target type/count, source support type, processing device, destination type, and abstract action templates.
• Y: one exact TextWorld command selected from the admissible commands.

The exact teacher prompt is implemented in code/prompts.py. See HINT_GENERATION.md for the complete construction and leakage-control rules.

Files

• Root adapter/tokenizer files: best LoRA checkpoint selected by validation loss.
• data/privileged_plan_sft.jsonl: 21,194 transition records with X, H, and Y.
• data/privileged_plan_sft.jsonl.manifest.json: split, class coverage, and leakage audit.
• data/privileged_plan_sft.jsonl.excluded_games.jsonl: 15 environment-timeout exclusions.
• code/: hint generation, final rewrite/audit, prompt, teacher SFT, and evaluation code.
• evaluation/: 59-task closed-loop summaries and trajectory replay audit.

Training data

The final data contains 3,538 successfully replayed games from 3,553 requested ALFWorld training games. Fifteen games (0.42%) were excluded after repeated TextWorld timeouts and are explicitly listed.

The split is game-level: 3,342 games / 20,006 records for training and 196 games / 1,188 records for validation. It is not a transition-random split.

Final leakage audit:

• H == Y: 0
• full gold command contained in H: 0
• digits or TextWorld instance IDs in H: 0
• gold command absent from admissible commands: 0
• episode-inconsistent hints: 0
• missing or mismatched source-support abstraction: 0

Teacher training

• Base model: Qwen/Qwen2.5-3B-Instruct
• Method: LoRA, rank 16, alpha 32, dropout 0.05
• Trainable parameters: 29,933,568
• Epochs: 2
• Seed: 42
• Global batch: 32 (8 GPUs × micro-batch 2 × accumulation 2)
• Learning rate: 2e-5, linear schedule, 10% warmup
• Maximum sequence length: 1,536
• Hardware: 8 × NVIDIA RTX 3090
• Final validation loss: 0.03002
• Best checkpoint: step 1,252, epoch 1

Ampere training uses eager attention because BF16 SDPA produced a reproducible non-finite backward gradient on a mixed-length batch. The uploaded adapter is from the stable eager-attention run.

Closed-loop evaluation

All conditions use the same 59 ALFWorld pilot games (35 valid_seen, 24 valid_unseen), seed 42, at most 50 environment steps, and greedy token-trie decoding constrained to the current admissible commands.

The pilot contains only pick-and-place tasks, so these success rates should not be presented as six-family ALFWorld results. Full trajectories were replayed independently: all 843 selected actions were admissible, all observations matched, and all 48 successes reproduced. The hint contained no digits or full selected action. Only 24.3% of teacher actions exactly matched the current oracle command, while 65.2% matched its abstract action template; the teacher often used alternative valid paths rather than copying an exact oracle trajectory.

Loading the adapter

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen2.5-3B-Instruct"
adapter_id = "TDSMike/ALF-qwen3B-privilege"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
tokenizer.pad_token = tokenizer.eos_token
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

For action selection, format the prompt with teacher_prompt(...) from code/prompts.py, and use the current environment's admissible commands. The released evaluation uses constrained greedy decoding; unconstrained free-form generation is not directly comparable.

Reproduction

1. Install the versions in requirements-reproduction.txt.
2. Prepare ALFWorld parquet files and game assets under verl_workspace/data/alfworld/.
3. Generate and audit hints using the commands in HINT_GENERATION.md.
4. Run code/sft_teacher.py with the configuration above.
5. Evaluate the immutable best checkpoint using code/eval_student.py --dataset_mode pilot --prompt_mode teacher.

Limitations

• H is privileged information derived from environment metadata and an expert plan; it is unavailable to a normal deployed agent.
• The 59-task reported evaluation covers only pick-and-place tasks.
• Results use one seed because of compute constraints.
• The adapter should be compared under identical candidate sets, prompts, splits, and decoding settings.