Llama-3-8B-Instruct-Iterative-SamPO

Model Card for Llama-3-8B-Instruct-Iterative-SamPO

This repository provides a fine-tuned version of Llama-3-8B-Instruct, using our proposed SamPO algorithm: Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence. We obey all licenses mentioned in llama3's work.

Performance

Evaluation Details

Five conditional benchmarks, using lm-evaluation-harness:

• GSM8K: 8-shot, report strict match
• IFEval: 3-shot, report instruction-level strict accuracy
• PiQA: 3-shot, report accuracy
• MMLU: 0-shot, report normalized accuracy
• TruthfulQA: 3-shot, report accuracy of single-true mc1 setting

One open-ended benchmark, using official alpaca_eval:

• AlpacaEval2: win rate (%) judged by GPT-4-turbo between the model's outputs vs. the GPT-4-turbo's response
• LC AlpacaEval2: length-debiased win rate (%) of AlpacaEval2
• Length in Tokens: the average output length of AlpacaEval2, calculated in tokens with Llama3's tokenizer

Input Format

The model is trained to use the following format:

<|start_header_id|>user<|end_header_id|>

{PROMPT}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

{Response}

Training hyperparameters

The following hyperparameters were used during DPO/SamPO training:

• DPO beta: 0.1
• learning_rate: 4e-7
• total_train_batch_size: 128
• optimizer: AdamW with beta1 0.9, beta2 0.999 and epsilon 1e-8
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• Weight Decay: 0.0
• num_epochs: 3.0
• Specifically add above input format over training samples