Pythia-31M-Chat-v1

A Pythia Chat Model of 31M Parameters

• Base model: EleutherAI/pythia-31m
• Availability in other ML formats:
  • GGUF: Felladrin/gguf-Pythia-31M-Chat-v1
  • ONNX: Felladrin/onnx-Pythia-31M-Chat-v1

Recommended prompt format

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Recommended inference parameters

penalty_alpha: 0.5
top_k: 2
repetition_penalty: 1.0016

Datasets and parameters used for training

SFTTrainer(
    model,
    train_dataset=train_dataset,
    dataset_text_field="text",
    eval_dataset=eval_dataset,
    max_seq_length=2048,
    packing=True,
    args=TrainingArguments(
        learning_rate=2e-6,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=16,
        lr_scheduler_type="cosine",
        num_train_epochs=1,
        logging_strategy="steps",
        save_strategy="steps",
        evaluation_strategy="steps",
        logging_steps=10,
        eval_steps=10,
        save_steps=10,
        warmup_steps=50,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        weight_decay=0.01,
        save_total_limit=10,
        neftune_noise_alpha=5,
    ),
    callbacks=[
        EarlyStoppingCallback(
            early_stopping_patience=3,
            early_stopping_threshold=0.005
        ),
    ],
)

DPOTrainer(
    model,
    beta=0.1,
    train_dataset=dataset,
    tokenizer=tokenizer,
    eval_dataset=eval_dataset,
    max_length=1536,
    max_prompt_length=1024,
    args=TrainingArguments(
        learning_rate=2e-6,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=1,
        lr_scheduler_type="cosine",
        num_train_epochs=1,
        logging_strategy="steps",
        save_strategy="steps",
        evaluation_strategy="steps",
        logging_steps=1,
        eval_steps=1,
        save_steps=1,
        warmup_steps=0,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        weight_decay=0.0,
        neftune_noise_alpha=5,
        remove_unused_columns=False,
    ),
    callbacks=[
        EarlyStoppingCallback(
            early_stopping_patience=3,
            early_stopping_threshold=0.005
        ),
    ],
)

Open LLM Leaderboard Evaluation Results

Detailed results can be found here