Downloads · 30 days
14
32% of all-time downloads
BigMistral-13b-GLUE_LORA
athirdpath/BigMistral-13b-GLUE_LORA
BigMistral-13b-GLUE_LORA is a machine learning model from athirdpath. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
All-time downloads
44
Public
Repo size
10.1 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin10.1 GB · 100%
At a glance
- Library
- peft
- Model type
- mistral
- Access
- Public
- Created
- Dec 3, 2023
- Updated
- Dec 3, 2023
- SHA
- 7ea930cf
Base models
Model card
From the Hugging Face model README
base_model: athirdpath/BigMistral-13b
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true
load_in_4bit: true
datasets:
- path: glueLORA2.jsonl
- type: alpaca
val_set_size: 0.07
adapter: qlora
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
lora_r: 512
lora_alpha: 32
lora_dropout: 0.04
lora_target_linear: true
gradient_accumulation_steps: 6
micro_batch_size: 3
eval_batch_size: 3
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.00005
bf16: true
gradient_checkpointing: true
flash_attention: true
warmup_steps: 10
weight_decay: 0.00001
- Library
- peft
- Type
- mistral
- Created
- Dec 3, 2023
- Updated
- Dec 3, 2023