Downloads · 30 days
7
25% of all-time downloads
llama3-gsm8k-lora
coastalcph/llama3-gsm8k-lora
llama3-gsm8k-lora is a machine learning model from coastalcph. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as other.
should probably proofread and complete it, then remove this comment. --
All-time downloads
28
Public
Parameters
8B
40.4 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors18.5 GB · 100%
At a glance
- Library
- peft
- License
- other
- Model type
- llama
- Access
- Public
- Created
- Jul 2, 2025
- Updated
- Jul 10, 2025
- SHA
- 033dc3e7
Base models
Datasets
Model card
From the Hugging Face model README
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
<details><summary>See axolotl config</summary>
axolotl version: 0.11.0.dev0
base_model: NousResearch/Meta-Llama-3-8B
# optionally might have model_type or tokenizer_type
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
load_in_8bit: true
load_in_4bit: false
datasets:
- path: cfierro/gsm8k_instr
type: alpaca
dataset_prepared_path: /workspace/axolotl-datasets/llama3-gsm8k
val_set_size: 0.05
output_dir: /workspace/axolotl-outputs/llama3-gsm8k-lora
sequence_len: 4096
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_modules_to_save:
- embed_tokens
- lm_head
wandb_project: weight-diff-ft
wandb_entity: cfierro
wandb_watch: all
wandb_name: llama3-gsm8k-lora
wandb_log_model: "false"
hub_model_id: cfierro/llama3-gsm8k-lora
gradient_accumulation_steps: 4
micro_batch_size: 2
max_steps: 1000 # 8k examples at most
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
bf16: auto
tf32: false
gradient_checkpointing: true
resume_from_checkpoint:
logging_steps: 1
flash_attention: true
warmup_steps: 10
early_stopping_patience: 5
eval_steps: 60 # bs=4*2 -> eval every 8*60=480 examples
save_steps: 60 # needed for config validation
save_total_limit: 1
load_best_model_at_end: true
weight_decay: 0.0
special_tokens:
pad_token: <|end_of_text|>
llama3-gsm8k-lora
This model is a fine-tuned version of NousResearch/Meta-Llama-3-8B on the cfierro/gsm8k_instr dataset. It achieves the following results on the evaluation set:
- Loss: 0.7904
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- training_steps: 1000
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0 | 0 | 1.2472 |
| 0.3777 | 1.5 | 60 | 0.5003 |
| 0.2575 | 3.0 | 120 | 0.5188 |
| 0.1077 | 4.5 | 180 | 0.6013 |
| 0.0657 | 6.0 | 240 | 0.6521 |
| 0.0199 | 7.5 | 300 | 0.7551 |
| 0.012 | 9.0 | 360 | 0.7904 |
Framework versions
- PEFT 0.15.2
- Transformers 4.52.4
- Pytorch 2.7.0+cu128
- Datasets 3.6.0
- Tokenizers 0.21.2
- Library
- peft
- Type
- llama
- License
- other
- Created
- Jul 2, 2025
- Updated
- Jul 10, 2025