fedagent-alfworld-ppo-uniform-1gpu-qwen2.5-1.5b

FedAgent · ALFWorld · PPO · uniform · Qwen2.5-1.5B-Instruct (single GPU) — unfinished run, round 54/70

Federated PPO on ALFWorld, 100 clients, 2 sampled per round, E=3 local epochs, planned 70 rounds, base_seed: 42. The run stopped at round 54 when its allocation ended; these are the checkpoints that survived it, published so the run can be resumed (or scored) from somewhere other than one cluster's scratch.

What is in here

PPO federates actor and critic, so a resume needs both roles of the same round; round 54 is the valid resume anchor. Round 39 is the peak of the curve and is kept because the rolling ckpt_latest overwrites a peak a few rounds later.

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AlexChen1997/fedagent-alfworld-ppo-uniform-1gpu-qwen2.5-1.5b"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="checkpoints/round_39_best/actor")
tok   = AutoTokenizer.from_pretrained(repo, subfolder="checkpoints/round_39_best/actor")

The curve (why round 39, not round 54)

In-loop validation is the first 64 valid_seen ALFWorld games, the same set every round (n=64), scored on the round's aggregated model.

The run peaked at 0.5938 @ round 39 and then decayed to 0.3906 by round 54. This is not a train/eval divergence: the training reward fell too (3.49 @ r40 → 2.89 @ r54), response length collapsed (177 → 87 tokens) and entropy went with it (0.24 → 0.09), i.e. the objective itself stopped working. That degradation is the reason the ref_anchor A/B exists: this run used the rolling KL reference (see below), and the open question is whether pinning the reference to the base model prevents the collapse.

Training setup

• Base model: Qwen/Qwen2.5-1.5B-Instruct.
• Federation: 100 clients, clients_per_round: 2, epochs_per_round: 3, total_rounds: 70 (reached 54), uniform partition, min_goals_per_client: 100.
• Algorithm: PPO (adv_estimator: gae), actor lr 1e-6, critic lr 1e-5, kl_loss_coef 0.01, low_var_kl, rollout.n=1, train batch 64, prompt/response 2048/512.
• KL reference = rolling (ref_anchor unset ⇒ the verl-0.8 default): verl builds actor and ref from the one model.path, and the driver moves each round's FedAvg'd weights through that key, so the reference follows the aggregate: J − 0.01·KL(π ‖ π_{r−1}), a per-round proximal term — not KL(π ‖ π_base). Anything compared against this run must use the same anchor.
• Environment: ALFWorld service, max_turns: 50, 64 train envs across 8 service replicas per client; validation as described above.
• Hardware / recipe: ONE H100-80GB (n_gpus_per_node: 1, FSDP world size 1), one of four single-GPU cells sharing a node; vLLM KV fraction 0.4, optimizer offload on. This is the single-GPU recipe, not the 4-GPU paper recipe — batch geometry matches, the parallelism does not, so timings and curves are not interchangeable with 4-GPU runs.
• Code: FedAgent (FedAgent_dev), single-GPU accelerated config tree; the run was resumed across several allocations and therefore spans commits ab4ecc3 → 2b1c535. ref_anchor was never set, so the rolling reference applies throughout.
• Config: fedagent/config/paper_accelerated_1gpu/uniform/Qwen2.5-1.5B-Instruct/main/ppo/ fed_alfworld_ppo_total-100_cl-per-rd-2_rd-70_ep-per-cl-3_min-goals-per-cl-100_p-uniform.yaml.

Caveats

• Unfinished: 54 of 70 rounds. Nothing here is a final model.
• One seed (42), one run; the in-loop metric is n=64, so a ±1 sd swing is ≈6 points of success.
• The checkpoints are the aggregated (FedAvg'd) models of their round, not any single client's.
• Raw per-round eval dumps, metrics and figures for this run live in github.com/canyuchen/fedagent_test (results/alfworld_ppo/, packaged at round 54).