Downloads · 30 days
13
12% of all-time downloads
shisa-ai/shisa-v2.1c-lfm2-350m-sft3
shisa-v2.1c-lfm2-350m-sft3 is a text generation model from shisa-ai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
13
12% of all-time downloads
All-time downloads
107
Public
Parameters
354M
709 MB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors709 MB · 99%
From the Hugging Face model README
axolotl version: 0.13.0.dev0
base_model: LiquidAI/LFM2-350M-ENJP-MT
chunked_cross_entropy: true
eot_tokens:
- "<|im_end|>"
datasets:
- path: chotto-20251010.sft.jsonl
type: chat_template
field_messages: conversations
message_property_mappings:
role: role
content: content
roles:
system:
- system
assistant:
- assistant
- gpt
- model
user:
- user
- human
roles_to_train: ["assistant"]
dataset_prepared_path: last_run_prepared_sft
output_dir: /data/outputs/shisa-v2.1c-lfm2-350m-sft2
sequence_len: 8192
sample_packing: true
flash_attention: true
pad_to_sequence_len: true
neftune_noise_alpha: 5
use_wandb: true
wandb_entity: augmxnt
wandb_project: liquid-hackathon-tokyo
wandb_name: "shisa-v2.1c-lfm2-350m-sft2"
# GBS = 128 / 8 GPU / 16 MBS / 1 GAS
gradient_accumulation_steps: 1
micro_batch_size: 16
num_epochs: 4
optimizer: adamw_torch_4bit
lr_scheduler: cosine
learning_rate: 6e-5 # 4.78 @ GBS=128
train_on_inputs: false
group_by_length: false
bf16: true
tf32: false
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
logging_steps: 1
warmup_ratio: 0.03
saves_per_epoch: 1
deepspeed: zero3_bf16.json
weight_decay: 1e-4
</details><br>
This model is a fine-tuned version of LiquidAI/LFM2-350M-ENJP-MT on the chotto-20251010.sft.jsonl dataset.
More information needed
More information needed
More information needed
The following hyperparameters were used during training: