Llama-3.1-8B-Tulu10pct-DPO-MAHALS

Llama-3.1-8B-Tulu10pct-DPO-MAHALS

Direct Preference Optimization (DPO) model trained on 10% of the Tulu-3 preference mixture for the MAHALS research project.

Model Details

Training Pipeline

meta-llama/Llama-3.1-8B
        ↓ SFT (10% Tulu-3 SFT mixture)
anonymousML123/Llama-3.1-8B-Tulu10pct-SFT-MAHALS
        ↓ DPO (10% Tulu-3 preference mixture)
anonymousML123/Llama-3.1-8B-Tulu10pct-DPO-MAHALS  ← This model

Inference Requirements

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "anonymousML123/Llama-3.1-8B-Tulu10pct-DPO-MAHALS"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [{"role": "user", "content": "What is machine learning?"}]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM Serving

vllm serve anonymousML123/Llama-3.1-8B-Tulu10pct-DPO-MAHALS --max_model_len=2048

Chat Template

This model uses the Tulu chat template:

<|user|>
Your question here
<|assistant|>
Model response<|endoftext|>

Training Configuration

DPO Training Metrics

Training Data

10% random sample from allenai/llama-3.1-tulu-3-8b-preference-mixture, which contains preference pairs for:

• Helpfulness
• Harmlessness
• Honesty
• Instruction following

Intended Use

This model is intended for research on multi-agent alignment and preference learning. It is part of the MAHALS (Multi-Agent Hierarchical Alignment) research project.

Limitations

• Trained on 10% of data (reduced capability vs full Tulu-3)
• English only
• May exhibit biases present in training data
• Not suitable for production without further evaluation

Citation

@misc{mahals2026,
  title={MAHALS: Multi-Agent Hierarchical Alignment},
  author={Anonymous},
  year={2026},
  note={Under review}
}

Acknowledgments

Built using AllenAI's open-instruct framework.