TinyLM-1-70M

TinyLM-1-70M

TinyLM-1-70M is a compact decoder-only Transformer language model designed for efficient instruction following and conversational AI. The model has approximately 72M parameters and has been fine-tuned using Supervised Fine-Tuning (SFT) on an instruction-response dataset to improve chat capabilities while maintaining a lightweight footprint suitable for local inference and resource-constrained environments.

Evaluation

All evaluations are zero-shot unless stated otherwise, and i used lm_eval to run them <img src="Benchmark.png">

Model Architecture & Hyperparameters

TinyLM-1-70M is built on a custom Gated Causal LM Transformer architecture with pre-normalization and SwiGLU activation functions:

---

Tokenizer & Chat Template

The model uses a custom Byte-Level BPE Tokenizer equipped with special tokens and a pre-configured Jinja2 chat_template for multi-turn conversations.

---

Training Configuration

---

🚀 Inference

# pip install torch transformers
                                                                             
import torch                                                                                            
from transformers import pipeline                                                                       
                                                                                                        
pipe = pipeline(                                                                                        
  "text-generation",                                                                                  
  model="Se00n00/TinyLM-1-70M",
  trust_remote_code = True                                                                            
)                                                                                                       
                                                                                                        
messages = [                                                                                            
  {"role": "system", "content": "You are a helpful AI assistant."},                                   
  {"role": "user", "content": "Explain artificial intelligence in simple terms."}                     
]                                                                                                       
                                                                                                        
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)       
                                                                                                        
result = pipe(                                                                                          
  prompt,                                                                                             
  max_new_tokens=120,                                                                                 
  do_sample=True,                                                                                     
  temperature=0.5,                                                                                    
  top_k=40,                                                                                           
  top_p=0.9                                                                                           
)                                                                                                       
                                                                                                        
print(result[0]['generated_text'])                                                                  

──────

Sample Outputs

System: You are a helpful AI assistant

User: What is the main concept of physics?

The main concept of physics is the idea that the laws of physics are governed by the laws of physics that govern the behavior of particles and forces . This idea is rooted in the idea that the laws of physics are not just a collection of laws , but are a fundamental aspect of the universe that govern s the behavior of matter , energy , and matter . The core idea of physics is that the laws of physics are governed by the laws of physics that govern the behavior of matter and energy . These laws are not a fixed set of laws , but rather a fundamental part of the universe that govern s the behavior of matter and energy . The laws of physics , which govern the behavior of matter and energy , are governed by the laws of physics that govern the behavior of matter and energy . These laws are not a fixed set of laws , but rather a fundamental aspect of the universe that govern s the behavior of matter and energy . The laws of physics , which govern the behavior of matter and energy , are governed by the laws of physics that govern the behavior of matter and energy . These laws are not a fixed set of laws , but rather a fundamental aspect of the universe that govern s the behavior of matter and energy . The laws of physics are not a fixed set of laws , but rather a fundamental part of the universe that govern s the behavior of matter and energy . The laws of physics are governed by the laws of physics that govern the behavior of matter and energy . These laws are not a fixed set of laws , but rather a fundamental part of the universe that govern s the behavior of matter and energy . The laws of physics are governed by the laws of physics that govern the behavior of matter and energy . These laws are not a fixed set of laws , but rather a fundamental part of the universe that govern s the behavior of matter and energy . The laws of physics are not a fixed set of laws , but rather a fundamental part of the universe that govern s the behavior of matter and energy . The laws of physics are governed by the laws of physics that govern the behavior of matter and energy . These laws are not a fixed set of laws , but rather a fundamental part of the universe that govern s the behavior of matter and energy . The laws of physics are not a fixed set of laws , but rather a fundamental part of the universe that govern s the behavior of matter and energy . The laws of physics are not a fixed set of laws , but rather a fundamental part of

System: You are a helpful AI assistant

User: Explain artificial intelligence in simple terms

The concept of artificial intelligence in simple terms is a fascinating and multif ace ted concept that has far - reaching implications in various fields . One of the most significant implications is that it can lead to the development of more advanced artificial intelligence , such as machine learning , artificial intelligence , and machine learning . AI systems can learn from data , identify patterns , and generate outputs , making them more accurate and reliable . This , in turn , can lead to the development of more efficient and effective AI systems . Another significant benefit of artificial intelligence is its ability to provide a more comprehensive and n uan ced understanding of the world . AI systems can learn from data , identify patterns , and make predictions , which can inform decision - making and decision - making . This , in turn , can lead to more effective decision - making , as AI systems can learn from data and make informed decisions based on patterns and trends . Furthermore , AI systems can also provide a more n uan ced and accurate understanding of the world . By analyzing vast amounts of data , AI systems can identify patterns and trends that might be difficult to discern from real - world data . This , in turn , can lead to more effective decision - making , as AI systems can learn from data and make predictions that are not always accurate . Furthermore , AI systems can also help to improve our ability to analyze and interpret large datasets . By analyzing vast amounts of data , AI systems can identify patterns and trends that might be difficult to discern from real - world data . This , in turn , can lead to more accurate and reliable data , as AI systems can make more accurate predictions and decisions . Lastly , AI systems can also be used to improve our ability to analyze and interpret large datasets . By analyzing large datasets , AI systems can identify patterns and trends that might be difficult to discern from real - world data . This , in turn , can lead to more accurate and reliable data , as AI systems can learn from data and make more accurate predictions . Overall , the concept of artificial intelligence in simple terms is a rich and multif ace ted concept that has far - reaching implications in various fields , including science , technology , and human understanding . While it has the potential to revolution ize the field of AI , it also raises important questions about the nature of intelligence , the impact of AI systems on society , and the role of AI in shaping human society .