SmolBabble-360m

SmolBabble-360m

This is SmolBabble-360m, the first ever 360 million parameter model finetuned on only children sentences. The base model is HuggingFaceTB/SmolLM-360M. This model was fine-tuned on a custom dataset of ~2,600 random simple sentences paired with completely unrelated single words (e.g., "appliance" -> "The man looked at her strangely through the window."). Because the training data contained zero actual instruction-response pairs, the dataset completely overwrote its instruct-tuning and learned only to output random declarative simple sentences. However, to some extent, it does occasionally contain the answer to questions in it's sentences, so not all of the base model was stripped away. See the Model Preview section.

Training Parameters

Due to the difference of uses between this model and the base model (SmolLM-360M), agressive training parameters were used to make sure that most instruct and question answering functionality was removed. Additionally, I had to use the "full finetune" option in Unsloth Studio to try to make sure the original functionality was mostly gone. The parameters used to train this are in the training_config.yaml file. It was directly exported from Unsloth Studio.

Model Preview

The table below has prompts and responses from SmolBabble-360m, which were generated during inference (Temperature: 0.7, Top-P: 0.95, Min-P: 0, System Instructions: None).

Quickstart

pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "benni-ben/SmolBabble-360m" 

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Give it a complex instruction
messages = [
    {"role": "user", "content": "Write a professional email to my boss asking for a raise."}
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(response)

Additional Info

Training Hardware

This model was trained on a MacBook Air M1 with 8GB of RAM. Luckily, since the model is so small, it can be run on mobile devices.

Contact

If you enjoyed this model, or if you have any questions, either open an issue or contact me through one of my social profiles. Thank you!