smollm2-135m-fineweb-edu2.7BT

Saiteja23/smollm2-135m-fineweb-edu2.7BT

Model Description

This is a 135M parameter language model based on the SmolLM2 architecture, trained on FineWeb-edu data.

Model Details

• Model Type: Causal Language Model (Decoder-only Transformer)
• Architecture: LLaMA-style with Grouped Query Attention
• Parameters: ~135M
• Training Data: 2.7B tokens from FineWeb-edu
• Base Model: HuggingFaceTB/SmolLM2-135M (tokenizer)
• License: Apache 2.0

Architecture Details

• Hidden Size: 576
• Attention Heads: 9
• Key-Value Heads: 3 (Grouped Query Attention)
• Layers: 30
• Vocabulary Size: 49,152
• Max Position Embeddings: 8,192
• Max Sequence Length: 512
• RoPE Theta: 100,000

Training Details

• Training Regime: Pretraining (no instruction tuning)
• Dataset: FineWeb-edu (sample-100BT)
• Training Tokens: 2.7B tokens
• Sequence Length: 512
• Batch Size: 128
• Learning Rate: 1e-3
• Optimizer: AdamW
• Precision: bfloat16

Usage

Text Generation

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")
model = AutoModelForCausalLM.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")

# Generate text
prompt = "The process of photosynthesis"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.8,
        top_k=50,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Custom Generation (using original model code)

# If you want to use the original model implementation
from model import ModelConfig, LlamaModel
from transformers import AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")
# Load using custom model class - see repository for details

Model Performance

This model is designed for:

• Text Continuation: Completing partial sentences or paragraphs
• Educational Content: Generating educational explanations and content
• Creative Writing: Short-form creative text generation
• General Language Modeling: Basic language understanding and generation

Limitations

• No Instruction Following: This is a pretrained model without instruction tuning
• No Safety Training: No specific safety or alignment training applied
• Educational Bias: Training data bias toward educational content
• Size Limitations: 135M parameters limit complex reasoning capabilities

Training Infrastructure

• Hardware: Single A100 80GB GPU
• Framework: PyTorch with custom training loop
• Distributed Training: Single GPU
• Mixed Precision: bfloat16

Citation

If you use this model, please cite:

@misc{smollm2-135m-fineweb-edu,
  title={SmolLM2-135M trained on FineWeb-edu},
  author={Your Name},
  year={2024},
  howpublished={\url{https://huggingface.co/Saiteja23/smollm2-135m-fineweb-edu2.7BT}},
}

Acknowledgments

• Base architecture inspired by SmolLM2 (HuggingFace)
• Training data from FineWeb-edu dataset
• Built with PyTorch and Hugging Face Transformers

---

This model was trained as part of a language model pretraining experiment. Use responsibly and be aware of potential biases in generated content.