Wiki-SmartBotLM-Instruct

Wiki-SmartBotLM-Instruct

A 270M parameter decoder-only Small Language Model (SLM) pretrained on English Wikipedia and instruction-tuned for general-purpose conversational AI.

---

Overview

Wiki-SmartBotLM-Instruct is a compact autoregressive language model developed entirely from scratch as an independent research project. The model was designed to explore the complete lifecycle of modern language model development—from pretraining through instruction tuning—using open datasets and modern transformer architectures.

Unlike models that rely on continued pretraining of existing foundation models, Wiki-SmartBotLM-Instruct was initialized with random weights and trained using a multi-stage pipeline.

The primary goals of this project were:

• Understand large language model training from first principles.
• Build a compact yet capable language model.
• Explore modern transformer architectures.
• Demonstrate the effectiveness of instruction tuning on a pretrained base model.

---

Model Details

---

Architecture

Wiki-SmartBotLM-Instruct incorporates several modern transformer improvements including:

• Rotary Positional Embeddings (RoPE)
• RMSNorm
• SwiGLU Feed Forward Networks
• Grouped Query Attention (GQA)
• Flash Attention
• Causal Self Attention
• Mixed Precision Training

---

Training Pipeline

The model was trained in two stages.

Stage 1 — Pretraining

The model was pretrained on the English Wikipedia corpus using next-token prediction.

Objectives:

• Learn English grammar
• Learn factual knowledge
• Learn encyclopedic writing style
• Learn semantic relationships
• Learn long-context language modeling

---

Stage 2 — Supervised Fine-Tuning (SFT)

Following pretraining, the model was instruction tuned using an open instruction-following dataset.

The objective of this stage was to improve:

• Instruction following
• Conversational ability
• Question answering
• Technical explanations
• General helpfulness

---

Training Data

Pretraining

• English Wikipedia (Hugging Face)

Supervised Fine-Tuning

• Alpaca Instruction Dataset

---

Capabilities

Wiki-SmartBotLM-Instruct can perform a variety of natural language tasks including:

• General Question Answering
• Technical Explanations
• Programming Assistance
• Educational Content Generation
• Text Summarization
• Brainstorming
• Long-form Text Generation
• Basic Reasoning
• Instruction Following

---

Benchmarks

The model was evaluated on the following text-based benchmarks:

• MMLU
• ARC Challenge
• HellaSwag
• Winogrande
• BoolQ

It shows despite the training difference between Gemma3 270M and WikiSmartBotLM, It does come close to Gemma 3 in some benchmarks.

---

Example

Prompt

Explain what artificial intelligence is.

Response

Artificial intelligence (AI) is a branch of computer science focused on building systems capable of performing tasks that typically require human intelligence. These tasks include reasoning, learning, planning, perception, natural language understanding, and decision making. Modern AI encompasses techniques such as machine learning, deep learning, computer vision, and reinforcement learning, and is widely used in applications ranging from recommendation systems to autonomous vehicles.

---

Intended Uses

Wiki-SmartBotLM-Instruct is intended for:

• Educational use
• AI research
• NLP experimentation
• Local inference
• Learning transformer architectures
• Small-scale deployment
• Prototype conversational assistants

---

Limitations

As a compact language model, Wiki-SmartBotLM-Instruct has several limitations.

• May hallucinate factual information.
• Limited reasoning compared to larger frontier models.
• Knowledge is limited to the pretraining corpus.
• May struggle with highly specialized or domain-specific questions.
• Performance may degrade on very long conversations.

---

Ethical Considerations

Wiki-SmartBotLM-Instruct is intended for research and educational purposes.

Although instruction tuning improves response quality, users should independently verify information before relying on generated content in high-stakes domains such as healthcare, finance, or legal advice.

---

Future Work

Future versions of Wiki-SmartBotLM aim to include:

• Improved factual grounding
• Larger instruction datasets
• Enhanced reasoning performance
• Better multilingual support
• Retrieval-Augmented Generation (RAG)
• Tool calling support
• Longer context windows

---

Citation

If you use Wiki-SmartBotLM-Instruct in your research, please cite:

@misc{wikismartbotlm2026,
  title={Wiki-SmartBotLM-Instruct: A 270M Parameter Decoder-Only Small Language Model},
  author={Pranav Upadhyaya},
  year={2026},
  howpublished={Hugging Face Model Repository}
}

---

Acknowledgements

This project was made possible by the open-source AI community.

Special thanks to:

• Hugging Face
• PyTorch
• English Wikipedia contributors
• Alpaca dataset contributors
• The open-source transformer research community

---

About the Project

Wiki-SmartBotLM-Instruct was developed as an independent research initiative to better understand modern language model development. From random initialization to pretraining and instruction tuning, every stage of the model was built to explore the practical engineering challenges of creating compact, efficient, and accessible language models.

Feedback, issues, and contributions are always welcome.