LFM2-1.2B-Pirate

LFM2-1.2B-Pirate

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

Fine-tuned model trained with Axolotl.

The process was very fast and efficient! It shows the fine-tuneability of LiquidAI/LFM-1.2B.

Thanks to @winglian for writing the entire code and making LFM2 compatible with Axolotl! ❤️

Example:

<|startoftext|><|im_start|>user
What is C. elegans?<|im_end|>
<|im_start|>assistant
Ahoy, matey!  You be talkin' 'bout the little scurvy dog, eh?  Well, shiver me timbers, that's a fine question! 

C. elegans, they say, be a tiny, scurvy dog, about the size of a thumbnail, but they're as smart as a kraken's tentacles and as clever as a pirate's treasure map. They be a little worm, but they've got a brain as big as a ship's mast and they can navigate through treacherous waters with ease. 

They be used in all sorts of experiments, like a pirate's treasure hunt, to learn more about how life works.  They be like a puzzle, a puzzle, and a treasure chest all rolled into one!  So, if you want to know more about these little creatures, just keep your eyes peeled, and maybe even get a bit closer, matey! 
<|im_end|>

🧩 Configuration

base_model: LiquidAI/LFM2-1.2B
xformers_attention: true
sample_packing: true
chunked_cross_entropy: true
learning_rate: 1e-5
sequence_len: 16384
micro_batch_size: 1
gradient_accumulation_steps: 2
gradient_checkpointing: true
optimizer: adamw_torch_8bit
lr_scheduler: cosine
warmup_ratio: 0.2
float16: true
bf16: false
max_grad_norm: 0.1
num_epochs: 3
saves_per_epoch: 1
logging_steps: 5
output_dir: ./outputs/lfm2-sft-pirate-rrr
chat_template: tokenizer_default
datasets:
  - path: winglian/pirate-ultrachat-10k
    type: chat_template
    split: train
eot_tokens:
  - "<|im_end|>"
dataloader_prefetch_factor: 8
dataloader_num_workers: 2
dataloader_pin_memory: true