mamba-2.8b-slimpj-OpenOrca_1ep

Model Card for mamba-2.8b-slimpj-OpenOrca_1ep

<!-- Provide a quick summary of what the model is/does. -->

This is a finetune of mamba-2.8b-slimpj for instruction following using the OpenOrca dataset.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is a finetune of the mamba reference model mamba-2.8b-slimpj from the paper https://arxiv.org/abs/2312.00752

It has been fine-tuned for instruction following using the OpenOrca dataset and training for 1 epoch.

• Model type: Mamba State Space Model (mamba_ssm)
• Finetuned from model: https://huggingface.co/state-spaces/mamba-2.8b-slimpj

Uses

This model is intended to evaluate fine-tuning results on mamba models.

Usage

Prompt structure

The prompt structure used in fine-tuning is alpaca format:

"### Human:\n%question%\n\n### AI response:\n%response%"

Training Details

Training Data

https://huggingface.co/datasets/Open-Orca/OpenOrca

Training Procedure

Trained using text-generation-webui with code from the mamba_ssm pull request.

Training Hyperparameters

• Training regime: Trained in bfloat16 with the following parameters:

{
  "trained_model_name": "mamba-2.8b-slimpj-OpenOrc_1ep",
  "save_steps": 500000.0,
  "micro_batch_size": 4,
  "batch_size": 128,
  "epochs": 1.0,
  "learning_rate": "3e-4",
  "lr_scheduler_type": "linear",
  "cutoff_len": 256,
  "dataset": "OpenOrca",
  "eval_dataset": "None",
  "format": "openorca-format",
  "warmup_steps": 100.0,
  "optimizer": "paged_adamw_8bit",
  "hard_cut_string": "\\n\\n\\n",
  "add_eos_token": false,
  "min_chars": 0.0,
}

Reported train_loss was 0.6762700151924311

Results

lm-evaluation-harness results for final model

mamba_ssm (pretrained=mamba-2.8b-slimpj-OpenOrca), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)

lm-evaluation-harness results after half epoch

mamba_ssm (pretrained=mamba-2.8b-slimpj-OpenOrca_1ep-checkpoints/checkpoint-500000), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)

Reference lm-evaluation-harness results for the base model mamba-2.8b-slimpj without fine-tuning

mamba_ssm (pretrained=mamba-2.8b-slimpj), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)

Summary

The models measured perplexity and accuracy got worse, but it's known that that can be an effect of fine-tuning. Perplexity and accuracy improved in the second half of the training, so it's likely that the inital worsening was caused by forcing a prompt structure onto the base model, which was trained only on unstructured text.

The answer quality as percieved by users is yet to be evaluated.

Environmental Impact

• Hardware Type: RTX 3090
• Hours used: 118