Downloads · 30 days
0
Ksgk-fy/maria_no_emoji_weak_SMaria-Meta-Llama-3.1-8B-Instruct
maria_no_emoji_weak_SMaria-Meta-Llama-3.1-8B-Instruct is a machine learning model from Ksgk-fy. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Llama3.1 seems to be sensitive to emoji, removing emoji from fine-tuning dataset help stabilize its behavior. I suspect this is related to its specific tokenization schemes. (Emoji's token might be mixed up with the e…
Downloads · 30 days
0
Access
Public
Updated Jul 25, 2024
Repo size
42 MB
Likes
0
Public
Click a slice to open those files.
.safetensors42 MB · 82%
From the Hugging Face model README
Llama3.1 seems to be sensitive to emoji, removing emoji from fine-tuning dataset help stabilize its behavior. I suspect this is related to its specific tokenization schemes. (Emoji's token might be mixed up with the eos_token). Plus, given the funny error with Llama3.1-8B-Instruct's tokenizer, it's normal that its finetuning process gets hurt like this.