ftb-sciworld-repro

ftb-sciworld-repro

A self-contained, runnable bundle for training five multi-turn on-policy distillation methods on ScienceWorld with a Qwen3-1.7B student and a Qwen3-32B teacher:

Everything needed to run is in here except the model weights, the task data and the Python dependencies, which scripts/setup.sh fetches.

Requirements

• 8 GPUs with ≥ 80 GB each, on one node. The teacher is dense Qwen3-32B at TP2 — roughly 32 GB of weights per device before any KV cache — so 40 GB cards cannot hold it at this parallelism. The split is 4 student rollout + 2 teacher
  • 2 FSDP trainer.
• Linux with CUDA and NCCL
• Python 3.10
• Java 17+ on PATH — ScienceWorld starts a JVM per episode
• ~80 GB free disk for models, plus ~4 GB per saved checkpoint (4 per run)

Quick start

hf download SeanWang0027/ftb-sciworld-repro --local-dir ftb-sciworld-repro
cd ftb-sciworld-repro

# If the node already has a working Trinity/TCOD environment:
bash scripts/setup.sh

# If it does not, also install the pinned dependency set:
bash scripts/setup.sh --with-deps

# Reuse models already on the node instead of downloading ~68 GB:
bash scripts/setup.sh --models /path/to/models     # expects Qwen3-1.7B, Qwen3-32B

# Then, one method at a time (each wants all 8 GPUs):
bash scripts/run.sh ftb
bash scripts/run.sh opd
bash scripts/run.sh guided_opd
bash scripts/run.sh tcod_f2b
bash scripts/run.sh tcod_b2f

Behind a scheduler: sbatch -A <acct> -p <partition> --export=ALL,METHOD=ftb scripts/run_slurm.sbatch.

(The Hub does not preserve the executable bit, so invoke the scripts as bash scripts/... — as above — rather than ./scripts/....)

setup.sh is idempotent — re-run it after a partial failure. It pip-installs the vendored tcod/ (with --no-deps unless you pass --with-deps), copies the FutureBridge overlay over it, verifies all five workflow classes import, puts the models under ./models, downloads the task split to ./data/scienceworld, and creates ./outputs/.

A run is 200 training steps, batch 16 tasks × up to 30 environment turns, with a checkpoint every 50 steps under outputs/checkpoints/FutureBridge-OPD/<run_name>/global_step_*/. Expect hours to a day per method; B2F and FTB are slowest (they boot two ScienceWorld JVMs per training task for gold-path replay).

What is upstream and what is not

This matters if you are reproducing published numbers.

Upstream, redistributed unmodified (both Apache-2.0, see NOTICE):

• tcod/ — kokolerk/TCOD at commit 17a8af2, exported with git archive
• overlay/ — ChenChiShui/FutureBridge-OPD source/ at commit bfcdf91
• configs/ftb.yaml — verbatim from that release

Added here: configs/{opd,guided_opd,tcod_b2f,tcod_f2b}.yaml, configs/bench.yaml.tmpl, everything in scripts/, this README, NOTICE.

The FutureBridge release ships ScienceWorld configs for FutureBridge and its ablations only. For the four baselines it ships the workflow implementations but no configuration — instead its configs/README.md gives a "Baseline workflow mappings" table saying to swap default_workflow_type and keep the paper settings. The four baseline YAMLs here are a reconstruction from that table: every shared setting is identical to ftb.yaml, and the deltas are the workflow class, the method's own workflow_args, run/buffer names, the monitor backend (tensorboard rather than wandb), and — for Guided-OPD only — the mix policy-loss block.

Judgement calls the upstream table does not determine, in case you want to change them:

Reporting results

The upstream protocol does not evaluate. Every released config sets eval_interval: 9999 and eval_on_startup: false, so the declared eval task set never fires, and no evaluation entry point is shipped. Its configs/README.md says to "run the configured evaluation task set once after training" but provides no mechanism.

What the release's own analysis code reads is the training rollout metric — analysis/plot_training_dynamics.py maps rollout/env_done/mean to the panel labelled "Completion Rate". So the faithful reproduction is: run the five methods and plot rollout/env_done/mean against step.

One caveat if you compare those curves across methods: the training rollout is not produced by the same policy in each arm. OPD's trajectories are 100% student; B2F's begin with a gold-action prefix; Guided-OPD's contain teacher turns with probability β (≈1 early in training); FTB's contain both. Early-training completion is therefore partly the teacher's or the gold path's work, not the student's.

scripts/eval.sh is provided as an addition, not a reproduction: it scores a run's checkpoints on the held-out 1,308-task test split through a single shared workflow, so the arms are measured under identical conditions with no teacher intervention. See the header of that script for why it forces one workflow for all five methods.

Either way the metric is env_done. There is no score metric anywhere, here or upstream: ScienceWorld's 0–100 partial credit is computed and stored on Experience.reward, which never reaches the metrics dictionary that the monitor aggregates. And env_done counts any terminating episode — ScienceWorld also terminates on failure, by driving the score negative — so it is not a clean success rate.

Known properties worth knowing before you interpret runs

These are properties of the upstream implementation, not bugs introduced here. None of them are patched in this bundle.

1. The prompt context accumulates, and overflow stops the agent. Four of the five workflows (all but Guided-OPD, which keeps a 10-turn window) send the entire growing chat history every turn, on top of a 2-step summary embedded in each user message. Measured with the Qwen3-1.7B tokenizer on real ScienceWorld rooms, the prompt crosses max_prompt_tokens: 10240 somewhere around turn 11 (object-rich room) to turn 17 (sparse room) of a 30-turn episode. Past that point the vLLM engine is not called at all: the framework returns a placeholder experience with an all-zero action mask, so the turn contributes no gradient, the parsed action is empty, and the agent stops acting for the rest of the episode. Grep a run log for Prompt was truncated to to count how often it happens.

Upstream hit a more severe version of this and fixed half of it: TCOD commit 17a8af2 ("use compact action representation to avoid token overflow") replaced a ~1162-item full action×object list with a ~43-item templates+objects pair, because single-step prompts were already exceeding the limit. That shrank the per-turn prompt; it did not change the accumulation.

2. The <think> instruction is inert. Both the system prompt and every user message require reasoning "enclosed within <think> </think> tags", but the configs run Qwen3 with enable_thinking: false, and that chat template pre-emits and closes an empty <think>\n\n</think>\n\n before the model writes a token. Across 11,264 recorded generation turns from three models in earlier measurements, <think> opened in 0.0% of them. The instruction costs tokens every turn and parse_action only ever reads <action>.

3. The full action×object list is hidden from the model but used to judge it. FTB's _is_valid_action checks a proposed action against get_valid_action_object_combinations() — the same ~1000-item list that commit 17a8af2 removed from the prompt. The student must compose a string that lands exactly in that set from two separate short lists, and a miss discards the whole bridge candidate.

4. FTB is B2F plus bridging. FutureBridgeScienceWorldWorkflow subclasses TCOD_b2f_scienceworld_workflow and overrides only _finalize_turn_responses. If you want to know whether FTB's gain comes from the bridge or from the B2F curriculum underneath it, tcod_b2f is the controlled comparison — not opd.

Troubleshooting

Credits

Method and implementation are the work of the upstream authors:

• TCOD — Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents, https://github.com/kokolerk/TCOD
• FutureBridge-OPD — Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation, https://github.com/ChenChiShui/FutureBridge-OPD

This bundle only packages them for reproducible execution and adds the missing baseline configurations. Please cite the upstream work.