Mixtress-135M

Mixtress 135M

Model Description

Mixtress 135M is a transformer model based upon the Mixtral architecture. It is the culmination of approximately 20 weeks of Kaggle free hours, and 67 twelve-hour training runs.

Training data

Mixtress was trained on a curated sampling of data from the following datasets:

• allenai/c4
• HuggingFaceFW/fineweb-edu
• togethercomputer/RedPajama-Data-V2
• Muennighoff/natural-instructions
• databricks/databricks-dolly-15k
• HuggingFaceTB/smollm-corpus
• open-phi/textbooks
• roneneldan/TinyStories

Training procedure

This model was trained for 2.15 billion tokens over 20,000 optimizer steps. It was trained as a masked autoregressive language model, using cross-entropy loss.

The final train loss was 1.941, validation loss was 2.206, and perplexity was 9.136.

Mixtress was pre-trained and fine-tuned simultaneously. Full reproduction code may be found at this URL, or in the Jupyter notebook in this repository.

Intended Use and Limitations

The model is best at what it was pretrained for, which is generating conversational text and answering questions from a prompt.

How to use

You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:

>>> from transformers import pipeline
>>> generator = pipeline('text-generation', model='UNSAFE/Mixtress-135M')
>>> generator("In a shocking finding, ", do_sample=True, temperature=0.7, min_length=50)

[{'generated_text': 'In a shocking finding, 20 years ago, U.S. President Donald Trump'}]

Eval results

All evaluations were done using the Pythia evaluation harness.

Scores

Join Us

If you would like to chat with us, please join the Discord server!