tulu-vista-repro

tulu-vista-repro

Llama-3.1-8B fine-tuned on the Tulu-3 SFT mixture using cross-entropy only (no ambient distillation). A reference / baseline reproduction of the allenai/Llama-3.1-Tulu-3-8B-SFT recipe, intended as a CE-only baseline for ambient knowledge-distillation ablations.

Model details

• Base model: meta-llama/Llama-3.1-8B
• Dataset: allenai/tulu-3-sft-mixture (~939k examples)
• Chat template: tulu (chat_template.jinja shipped with the model)
• Tokenizer: base Llama-3.1 tokenizer with a single <pad> token appended; vocab 128256 -> 128257, padded to 128264 for tensor-core alignment
• Precision: bf16 mixed precision

Training

• Optimizer: AdamW (decoupled)
• Learning rate: 5e-6, linear schedule, warmup ratio 0.03, weight decay 0.0
• Sequence length: 4096
• Epochs: 2 (--num_train_epochs 2)
• Final checkpoint: step 14613 of 14614
• Effective batch size: 128 sequences (per-device batch 1 × gradient accumulation 4 × 32 GPUs)
• Distributed: DeepSpeed ZeRO-2, fp32 gradient communication, 32-way data parallel on 32× NVIDIA H200 GPUs
• Framework: open-instruct + accelerate + DeepSpeed

Evaluation

Scores via olmes HF backend, dtype=bfloat16, tulu chat template, max_length=4096, parallelize=false.

For comparison, the official allenai/Llama-3.1-Tulu-3-8B-SFT reproduces at 61.44 on the same 10-task mean. This CE-only Vista repro is ~3 points below, with the gap concentrated on IFEval, HumanEval, MMLU, and Minerva — consistent with the official recipe using better-tuned hyperparameters, possibly more careful data ordering, and/or longer training.

Intended use

This checkpoint is released as a reference baseline for ambient-KD ablations on Llama-3.1-8B + Tulu-3. It is not intended as a drop-in replacement for the official Tulu-3 SFT model.

License

Llama 3.1 Community License (inherited from meta-llama/Llama-3.1-8B). The Tulu-3 SFT mixture carries its own per-source dataset licenses; refer to allenai/tulu-3-sft-mixture for details.

Citation

If you use this model, please cite the Tulu-3 paper:

@article{lambert2024tulu3,
  title={Tulu 3: Pushing Frontiers in Open Language Model Post-Training},
  author={Lambert, Nathan and others},
  journal={arXiv preprint},
  year={2024}
}