Downloads · 30 days
6
4% of all-time downloads
shisa-v2-dev/ablation-135-geniac.gbs128.2e6-shisa-v2-llama-3.1-8b
ablation-135-geniac.gbs128.2e6-shisa-v2-llama-3.1-8b is a text generation model from shisa-v2-dev. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.1.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
6
4% of all-time downloads
All-time downloads
140
Public
Parameters
8B
16.1 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors16.1 GB · 100%
From the Hugging Face model README
axolotl version: 0.8.0.dev0
# train w/ shisa-ai/shisa-v1-athenev2-reannotated-filtered
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
# User Liger
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true
chat_template: llama3
datasets:
- path: shisa-ai/shisa-v2-best-of-n-athenev2-tulu70b-llama33-only-no-sysprompt
type: chat_template
field_messages: conversations
message_property_mappings:
role: from
content: value
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
roles_to_train: ["assistant"]
- path: shisa-ai/shisa-v2-roleplaying-sft
type: chat_template
field_messages: conversations
message_property_mappings:
role: role
content: content
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
roles_to_train: ["assistant"]
- path: shisa-ai/translation_expanded_master_set_filtered
split: train[:25%]
type: chat_template
field_messages: conversations
message_property_mappings:
role: role
content: content
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
roles_to_train: ["assistant"]
- path: shisa-ai/rewild-set
split: train[:5%]
type: chat_template
field_messages: conversations
message_property_mappings:
role: role
content: content
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
roles_to_train: ["assistant"]
- path: shisa-ai/magpie-ultra-set
split: train[:8%]
type: chat_template
field_messages: conversations
message_property_mappings:
role: role
content: content
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
roles_to_train: ["assistant"]
- path: shisa-ai/magpie-advanced-questions-set
split: train[:8%]
type: chat_template
field_messages: conversations
message_property_mappings:
role: role
content: content
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
roles_to_train: ["assistant"]
- path: shisa-ai/japan-magpie-set
split: train
type: chat_template
field_messages: conversations
message_property_mappings:
role: role
content: content
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
roles_to_train: ["assistant"]
- path: shisa-ai/ko_dataset_conversations
split: train
type: chat_template
field_messages: conversations
message_property_mappings:
role: from
content: value
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
- path: shisa-ai/tmmluplus_sim
split: train
type: chat_template
field_messages: conversations
message_property_mappings:
role: from
content: value
roles:
system:
- system
assistant:
- gpt
- model
- assistant
user:
- human
- user
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./outputs/ablation-135-geniac.gbs128.2e6-shisa-v2-llama-3.1-8b
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
# marginal difference
neftune_noise_alpha: 5
use_wandb: true
wandb_project: shisa-v2
wandb_entity: augmxnt
wandb_name: ablation-135-geniac.gbs128.2e6-shisa-v2-llama-3.1-8b
gradient_accumulation_steps: 2
micro_batch_size: 4
num_epochs: 3
optimizer: paged_adamw_8bit
lr_scheduler: linear
learning_rate: 2e-6
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 0
save_total_limit: 1 # Only store a single checkpoint
debug:
deepspeed: zero3_bf16.json
weight_decay: 1e-4
fsdp:
fsdp_config:
special_tokens:
pad_token: <|end_of_text|>
</details><br>
This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the shisa-ai/shisa-v2-best-of-n-athenev2-tulu70b-llama33-only-no-sysprompt, the shisa-ai/shisa-v2-roleplaying-sft, the shisa-ai/translation_expanded_master_set_filtered, the shisa-ai/rewild-set, the shisa-ai/magpie-ultra-set, the shisa-ai/magpie-advanced-questions-set, the shisa-ai/japan-magpie-set, the shisa-ai/ko_dataset_conversations and the shisa-ai/tmmluplus_sim datasets. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.2885 | 0.0021 | 1 | 1.3167 |
| 1.0369 | 0.5 | 243 | 1.0408 |
| 1.0075 | 1.0 | 486 | 0.9979 |
| 0.9414 | 1.5 | 729 | 0.9773 |
| 0.9372 | 2.0 | 972 | 0.9644 |
| 0.9351 | 2.5 | 1215 | 0.9587 |
| 0.9297 | 3.0 | 1458 | 0.9561 |