Llama-TinyStories

Llama-TinyStories

This model is for educational purposes only. If you are looking for the actual TinyStories model to use, refer to the TinyStories-33M by Eldan & Li.

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/68690383415ee2226b7a8973/UEmMB_poDMhngMtmJt4Sx.png"> </p>

Reproduction of the TinyStories (Eldan & Li, 2023) model using a LLaMA-style architecture, trained on ~1B tokens with nanotron.

Read the accompanying blog post for a detailed walkthrough of the training process.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model = AutoModelForCausalLM.from_pretrained('vinothkumarn/Llama-TinyStories')
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-125M")
prompt = "Once upon a time there was"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Generate completion
output = model.generate(input_ids, max_length = 256, num_beams=1)

# Decode the completion
output_text = tokenizer.decode(output[0], skip_special_tokens=True)

# Print the generated text
print(output_text)

Model Details

Limitations

• English only, children's story domain
• Educational/research use — not for production
• Can be repetitive on long generations (only 4 layers)

License

MIT. Training data licensed under CDLA-Sharing-1.0.

Citation

@article{eldan2023tinystories,
  title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
  author={Eldan, Ronen and Li, Yuanzhi},
  journal={arXiv preprint arXiv:2305.07759},
  year={2023}
}