2026.AP.cpt_dense

2026.AP.cpt_dense

Dense-baseline continued pretraining (CPT) run: Qwen2.5-3B trained on fineweb-edu with NO pruning mask, as the dense control arm of the replication.

Part of a replication of Apple's Instruction-Following Pruning for Large Language Models (arXiv:2501.02086) — "AP" = apple-paper-replicate, package ifpruning in Sid-MB/mats_exploration under code/apple-paper-replicate/ (branch introspection-causal-test, merged to main at a76965b).

Architecture

• LLM: Qwen2.5-3B (Qwen2ForCausalLM, 36 layers, d_ffn 11008)
• Predictor backbone: Qwen2.5-0.5B (Qwen2Model) + 2-layer MLP head (head.pt) producing per-layer FFN importance scores [36, 11008]; per-row SoftTopK selects t_ffn=1536 of 11008 FFN units (~1B activated params). Dense-baseline runs train the same LLM without masking.

Contents

• checkpoints/step_3000/pytorch_model_fsdp_0/ — FSDP2 SHARDED_STATE_DICT model weights (llm + predictor_backbone + head)
• checkpoints/step_3000/optimizer_0/ — optimizer state (for exact training resumption)
• checkpoints/step_3000/random_states_*.pkl, scheduler.bin — RNG/scheduler state
• model/ — exported final model (llm/, predictor_backbone/, head.pt) from the FSDP export step

Performance

No evaluations were run on this checkpoint. The project reached "scaffold + smoke test + this training grid" before being paused (see code/mats_exploration/everything we learned.md); logs/eval/ is empty and no eval_results directory exists. The only training-quality signal is the loss curves in the wandb runs below.

Reproduction

From code/ in the mats_exploration repo (paths as of June 2026; IFP_ROOT=/nlp/scr/siddharth/apple-paper-replicate set in slurm/_common.sh):

sbatch apple-paper-replicate/slurm/train.sbatch apple-paper-replicate/configs/presets/cpt_dense.yaml

which runs (8 GPUs, accelerate FSDP2 full-shard bf16, SHARDED_STATE_DICT):

srun uv run accelerate launch --config_file apple-paper-replicate/configs/accelerate_fsdp8.yaml \
    -m ifpruning.train --config apple-paper-replicate/configs/presets/cpt_dense.yaml \
    --ckpt-root $IFP_ROOT/ckpts --out-root $IFP_ROOT/models

Preset: configs/presets/cpt_dense.yaml. Data: fineweb-edu sample-10BT, streaming, seq_len 4096. Seed 0. Slurm job 15878286 (jagupard39, 8 GPUs).

Weights & Biases

• https://wandb.ai/siddharth-stanford/ifpruning-cpt/runs/qndp904y

Cluster paths (Stanford NLP)

• Training log: /nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/logs/train/cpt_dense_15878286.out
• Original checkpoint dir: /nlp/scr2/siddharth/apple-paper-replicate/ckpts/cpt_dense (deleted after this upload was verified; this repo is now the only copy)
• Code, configs, paper PDF, research notes: /nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/
• Research note with the full run grid: .../research-notes/2026-06-12_setup-and-smoke.md