Downloads · 30 days
9
12% of all-time downloads
nicoboss/Mantella-Skyrim-Llama-3-8B-Uncensored-Lora
Mantella-Skyrim-Llama-3-8B-Uncensored-Lora is a text generation model from nicoboss. Use it when you need the model to write or continue text. It is set up for peft.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
9
12% of all-time downloads
All-time downloads
73
Public
Repo size
21.5 GB
Likes
1
Public
Click a slice to open those files.
.pt18.8 GB · 86%
From the Hugging Face model README
axolotl version: 0.12.0.dev0
base_model: /pool16_2/Mantella-Skyrim-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
# for use with fft to only train on language model layers
# unfrozen_parameters:
# - model.language_model.*
# - lm_head
# - embed_tokens
load_in_8bit: false
load_in_4bit: false
# these 3 lines are needed for now to handle vision chat templates w images
#skip_prepare_dataset: true
#remove_unused_columns: false
#sample_packing: false
# gemma3 doesn't seem to play nice with ddp
#ddp_find_unused_parameters: true
chat_template: llama3
datasets:
- path: /root/Uncensored_Reasoner_Small_Chat.json
type: chat_template
field_messages: messages
dataset_prepared_path: last_run_prepared_Mantella-Skyrim-Llama-3-8B-uncensored_final
val_set_size: 0.01
output_dir: ./HDD/Mantella-Skyrim-Llama-3-8B-uncensored_final
adapter: lora
# lora_model_dir:
peft_use_rslora: true
sequence_len: 5400
pad_to_sequence_len: false
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 8
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 0.00004
bf16: auto
fp16:
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
auto_resume_from_checkpoints: true
logging_steps: 1
flash_attention: true
#eager_attention: true
warmup_steps: 50
evals_per_epoch: 2
eval_max_new_tokens: 128
saves_per_epoch: 2
save_total_limit: 100
debug:
weight_decay: 0.0
deepspeed: deepspeed_configs/zero1.json
special_tokens:
pad_token: <|end_of_text|>
</details><br>
This model was trained from scratch on the /root/Uncensored_Reasoner_Small_Chat.json dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0 | 0 | 1.8614 |
| 1.0431 | 0.5 | 281 | 1.1419 |
| 1.206 | 1.0 | 562 | 1.0708 |
| 1.061 | 1.5 | 843 | 1.0362 |
| 0.7878 | 2.0 | 1124 | 1.0065 |
| 0.8042 | 2.5 | 1405 | 0.9940 |
| 0.673 | 3.0 | 1686 | 0.9824 |
| 0.6364 | 3.5 | 1967 | 1.0085 |
| 0.5152 | 4.0 | 2248 | 0.9915 |
| 0.4905 | 4.5 | 2529 | 1.0410 |
| 0.5221 | 5.0 | 2810 | 1.0166 |
| 0.2905 | 5.5 | 3091 | 1.0783 |
| 0.3028 | 6.0 | 3372 | 1.0636 |
| 0.2572 | 6.5 | 3653 | 1.1089 |
| 0.3408 | 7.0 | 3934 | 1.1059 |
| 0.2769 | 7.5 | 4215 | 1.1322 |