Downloads · 30 days
15
5% of all-time downloads
mistral-7b-cot-neftune-v0.1
YeonwooSung/mistral-7b-cot-neftune-v0.1
mistral-7b-cot-neftune-v0.1 is a text generation model from YeonwooSung. Use it when you need the model to write or continue text. It is set up for transformers.
The Mistral-7B-v0.1 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. This model uses peft finetuning with NEFTune for robustness.
All-time downloads
321
Public
Parameters
7.4B
4.5 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors4.5 GB · 100%
Parameter types
How the weights are stored.
U87B · 95%
Try a prompt
Model card
From the Hugging Face model README
Model Card for Model ID
The Mistral-7B-v0.1 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. This model uses peft finetuning with NEFTune for robustness.
Model Details
Model Description
This model is a finetuned model of the mistralai/Mistral-7B-v0.1.
Training Details
Training Data
This model is finetuned with kaist-ai/CoT-Collection.
Training Procedure
This model trained with SFT trainer and NEFTune method. (According to the paper, NEFTune adds noise to the embedding vectors during training)
Training Hyperparameters
- lora alpha: 16
- lora r: 64
- lora dropout: 0.05
- max sequence length: 4096
- learning rate: 2e-4
- max_grad_norm: 0.3
- weight_decay: 0.001
- gradient checkpoint: True
- optim: paged_adamw_32bit
- use_bf16: True
- use_4bit: True
- use_nested_quant: False
- bnb_4bit_compute_dtype: float16
- bnb_4bit_quant_type: nf4
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- Created
- Mar 3, 2024
- Updated
- Mar 3, 2024