2026.AP.sft_dense

2026.AP.sft_dense

Dense-baseline SFT run: initialized from the cpt_dense model (init_from models/cpt_dense) and fine-tuned on tulu-3-sft-mixture, as the dense control arm.

Part of a replication of Apple's Instruction-Following Pruning for Large Language Models (arXiv:2501.02086) — "AP" = apple-paper-replicate, package ifpruning in Sid-MB/mats_exploration under code/apple-paper-replicate/ (branch introspection-causal-test, merged to main at a76965b).

Architecture

• LLM: Qwen2.5-3B (Qwen2ForCausalLM, 36 layers, d_ffn 11008)
• Predictor backbone: Qwen2.5-0.5B (Qwen2Model) + 2-layer MLP head (head.pt) producing per-layer FFN importance scores [36, 11008]; per-row SoftTopK selects t_ffn=1536 of 11008 FFN units (~1B activated params). Dense-baseline runs train the same LLM without masking.

Contents

• checkpoints/step_1000/pytorch_model_fsdp_0/ — FSDP2 SHARDED_STATE_DICT model weights (llm + predictor_backbone + head)
• checkpoints/step_1000/optimizer_0/ — optimizer state (for exact training resumption)
• checkpoints/step_1000/random_states_*.pkl, scheduler.bin — RNG/scheduler state

Performance

No evaluations were run on this checkpoint. The project reached "scaffold + smoke test + this training grid" before being paused (see code/mats_exploration/everything we learned.md); logs/eval/ is empty and no eval_results directory exists. The only training-quality signal is the loss curves in the wandb runs below.

Reproduction

From code/ in the mats_exploration repo (paths as of June 2026; IFP_ROOT=/nlp/scr/siddharth/apple-paper-replicate set in slurm/_common.sh):

sbatch apple-paper-replicate/slurm/train.sbatch apple-paper-replicate/configs/presets/sft_dense.yaml

which runs (8 GPUs, accelerate FSDP2 full-shard bf16, SHARDED_STATE_DICT):

srun uv run accelerate launch --config_file apple-paper-replicate/configs/accelerate_fsdp8.yaml \
    -m ifpruning.train --config apple-paper-replicate/configs/presets/sft_dense.yaml \
    --ckpt-root $IFP_ROOT/ckpts --out-root $IFP_ROOT/models

Preset: configs/presets/sft_dense.yaml. Data: allenai/tulu-3-sft-mixture. Seed 0. Slurm job 15878289 (jagupard39, 8 GPUs, afterok:15878286).

Weights & Biases

• https://wandb.ai/siddharth-stanford/ifpruning-sft/runs/v3objrks

Cluster paths (Stanford NLP)

• Training log: /nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/logs/train/sft_dense_15878289.out
• Original checkpoint dir: /nlp/scr2/siddharth/apple-paper-replicate/ckpts/sft_dense (deleted after this upload was verified; this repo is now the only copy)
• Code, configs, paper PDF, research notes: /nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/
• Research note with the full run grid: .../research-notes/2026-06-12_setup-and-smoke.md