Downloads · 30 days
9
4% of all-time downloads
Mistral-7b-tokens4b-v1
kirv/Mistral-7b-tokens4b-v1
Mistral-7b-tokens4b-v1 is a machine learning model from kirv. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for torchtune. The card lists the license as apache-2.0.
Модель mistralai/Mistral-7B-v0.1, обучение всех слоев с ~4млрд токенов из датасета.
All-time downloads
233
Public
Parameters
7.2B
43.5 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors14.5 GB · 100%
At a glance
- Library
- torchtune
- License
- apache-2.0
- Model type
- mistral
- Access
- Public
- Created
- May 5, 2024
- Updated
- May 5, 2024
- SHA
- b28d10d0
Base models
Datasets
Model card
From the Hugging Face model README
Модель mistralai/Mistral-7B-v0.1, обучение всех слоев с ~4млрд токенов из датасета.
130 часов 2xTesla H100.
batch_size: 20
epochs: 1
optimizer:
_component_: torch.optim.AdamW
lr: 5e-6
weight_decay: 0.01
loss:
_component_: torch.nn.CrossEntropyLoss
max_steps_per_epoch: null
gradient_accumulation_steps: 5
Размер последовательности 1024 токенов.
Loss curve
По https://github.com/NLP-Core-Team/mmlu_ru
Квантизация в 4b: accuracy_total=41.86218134391028
- Library
- torchtune
- Type
- mistral
- License
- apache-2.0
- Languages
- ru, en
- Created
- May 5, 2024
- Updated
- May 5, 2024