smollm-135m-genqa

🌟 SmolLM-135M SQuAD v1.1 Generative Question Answering Model

Fine-tuned SmolLM-135M-Instruct on SQuAD v1.1 for English generative question answering.

This model takes a context paragraph and a question and generates a natural-language answer based on the text.
Unlike extractive QA models (e.g., BERT), this model does not select spans—it produces free-form answers using a causal language modeling head.

It is intended as a compact, educational example of fine-tuning a small (~135M parameter) generative LM for QA on Google Colab and deploying it on Hugging Face.

---

🔎 Use Cases

• Educational demos of generative question answering
• Small QA systems over short English paragraphs
• Teaching / learning how to:
  • convert extractive data to generative format
  • fine-tune LMs with Trainer
  • publish models + Spaces on Hugging Face

⚠️ Not intended for production-critical use (medical/legal/financial advice, etc.).

---

🧠 Model Details

• Base model: HuggingFaceTB/SmolLM-135M-Instruct
• Architecture: Decoder-only Transformer with causal LM head
• Parameters: ~135M
• Task: Generative Question Answering
• Language: English
• Author: omarbayoumi2
• Training platform: Google Colab (free GPU)

The model generates answers instead of choosing spans directly from the context.

---

📚 Training Data

• Dataset: SQuAD v1.1 (rajpurkar/squad)
• Domain: Wikipedia articles (encyclopedic text)

Each example was converted into a generative prompt:

• context: paragraph
• question: question string
• answers:

Only answer tokens contribute to the training loss (prompt tokens masked with -100).

---

⚙️ Training Configuration

Fine-tuning was performed with the Hugging Face Trainer API.

• Objective: Causal LM
• Epochs: 2
• Learning rate: 2e-4
• Batch size: 8 (train), 8 (eval)
• Max sequence length: 256 tokens
• Weight decay: 0.01
• Padding strategy: Max length
• Mixed precision: FP16 (when GPU supports it)

Loss is computed over answer tokens only.

---

📊 Evaluation

Evaluation was performed on a subset of the SQuAD v1.1 validation set using extractive metrics:

• Exact Match (EM): strict text match
• F1 score: token-level overlap

Because outputs are generative rather than span-based, metrics may underestimate correctness.

This model is optimized for educational usage, not leaderboard performance.

---

🚀 How to Use

1. With Transformers pipeline

from transformers import pipeline

qa = pipeline(
    "text-generation",
    model="omarbayoumi2/smollm-135m-genqa",
)

context = "BERT is a language representation model developed by researchers at Google."
question = "Who developed BERT?"

prompt = f"Context: {context}\n\nQuestion: {question}\n\nAnswer:"

result = qa(prompt, max_new_tokens=64, do_sample=False)[0]["generated_text"]
print(result)