Downloads · 30 days
8
4% of all-time downloads
shisa-ai/shisa-v1-llama3-8b.lr-5e6
shisa-v1-llama3-8b.lr-5e6 is a text generation model from shisa-ai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.
The was part of some LR ablations. It's not bad but you should probably prefer 8e-6
Downloads · 30 days
8
4% of all-time downloads
All-time downloads
213
Public
Parameters
8B
16.1 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors16.1 GB · 100%
From the Hugging Face model README
The was part of some LR ablations. It's not bad but you should probably prefer 8e-6
I ran the tests for 2 runs just to try to lower variance. These are all using temp 0.2, min_p 0.1, freq penalty 0.5
| Model | AVG Score | ELYZA100 | JA MT-Bench | Rakuda | Tengu-Bench | JA Char % |
|---|---|---|---|---|---|---|
| shisa-v1-llama3-8b.lr-2e4 | 3.97 | 4.60 | 4.54 | 3.33 | 3.42 | 92.42% |
| shisa-v1-llama3-8b.lr-5e5 | 5.73 | 6.28 | 6.45 | 5.37 | 4.81 | 90.93% |
| shisa-v1-llama3-8b (2e5 avg) | 6.33 | 6.51 | 6.66 | 6.68 | 5.48 | 91.51% |
| shisa-v1-llama3-8b.8e6 | 6.59 | 6.67 | 6.95 | 7.05 | 5.68 | 91.30% |
| shisa-v1-llama3-8b.5e6 | 6.42 | 6.33 | 6.76 | 7.15 | 5.45 | 91.56% |
| shisa-v1-llama3-8b.2e6 | 6.31 | 6.26 | 6.88 | 6.73 | 5.38 | 92.00% |
axolotl version: 0.4.0
base_model: meta-llama/Meta-Llama-3-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
chat_template: llama3
datasets:
- path: augmxnt/ultra-orca-boros-en-ja-v1
type: sharegpt
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./outputs/lr-5e6
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
use_wandb: true
wandb_project: shisa-v2
wandb_entity: augmxnt
wandb_name: shisa-v1-llama3-8b.lr-5e6
gradient_accumulation_steps: 8
micro_batch_size: 1
num_epochs: 3
optimizer: paged_adamw_8bit
lr_scheduler: linear
learning_rate: 5e-6
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 0
debug:
deepspeed: axolotl/deepspeed_configs/zero3_bf16.json
weight_decay: 0.00
fsdp:
fsdp_config:
special_tokens:
pad_token: <|end_of_text|>
</details><br>
This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the None dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.3951 | 0.0064 | 1 | 0.8645 |
| 0.891 | 0.5020 | 79 | 0.5705 |
| 0.8575 | 1.0040 | 158 | 0.5243 |
| 0.7296 | 1.4853 | 237 | 0.5079 |
| 0.7068 | 1.9873 | 316 | 0.4976 |
| 0.6618 | 2.4694 | 395 | 0.5020 |