Downloads · 30 days
937
1% of all-time downloads
TinyMistral-248M
Locutusque/TinyMistral-248M
TinyMistral-248M is a text generation model from Locutusque. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A pre-trained language model, based on the Mistral 7B model, has been scaled down to approximately 248 million parameters. This model has been trained on 7,488,000 examples. This model isn't intended for direct use bu…
All-time downloads
100K
Public
Parameters
248M
6.4 GB on disk
Likes
50
Public
Repo makeup
Click a slice to open those files.
.bin992 MB · 50%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mistral
- Access
- Public
- Created
- Nov 14, 2023
- Updated
- May 9, 2024
- SHA
- e84fb529
Try a prompt
Model card
From the Hugging Face model README
A pre-trained language model, based on the Mistral 7B model, has been scaled down to approximately 248 million parameters. This model has been trained on 7,488,000 examples. This model isn't intended for direct use but for fine-tuning on a downstream task. This model should have a context length of around 32,768 tokens. Safe serialization has been removed due to issues saving model weights.
During evaluation on InstructMix, this model achieved an average perplexity score of 6.3. More epochs are planned for this model on different datasets.
Open LLM Leaderboard Evaluation Results (outdated)
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 24.18 |
| ARC (25-shot) | 20.82 |
| HellaSwag (10-shot) | 26.98 |
| MMLU (5-shot) | 23.11 |
| TruthfulQA (0-shot) | 46.89 |
| Winogrande (5-shot) | 50.75 |
| GSM8K (5-shot) | 0.0 |
| DROP (3-shot) | 0.74 |
The purpose of this model is to prove that trillion-scale datasets are not needed to pretrain a language model. As a result of needing small datasets, this model was pretrained on a single GPU (Titan V).
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- apache-2.0
- Languages
- en
- Created
- Nov 14, 2023
- Updated
- May 9, 2024