Sutra-Instruct-350M

Sutra-Instruct-350M

Sutra-Instruct-350M is a custom-built, 350-million parameter causal language model trained using nanaGPT architecture.

🧠 Model Architecture & Details

• Architecture: Custom nanoGPT-based Transformer
• Parameter Count: 350M
• Format: safetensors
• Embeddings: Tied (lm_head and wte share memory)
• Creator: Abhiray

📚 Training Pipeline

This model was not fine-tuned from an existing corporate base model (like Llama or Mistral). Its brain was initialized from absolute zero and trained through a rigorous two-phase pipeline:

Phase 1: Pre-Training (The Foundation) The base logic was built by streaming a highly curated mix of academic and coding datasets:

• HuggingFaceFW/fineweb-edu (High-level English and academic structure)
• open-web-math/open-web-math (Mathematical logic and formatting)
• bigcode/starcoderdata (Python syntax and code structure)
• roneneldan/TinyStories (Basic grammar and narrative flow)

Phase 2: Supervised Fine-Tuning (SFT) Once the model learned how to speak, it was fine-tuned using the yahma/alpaca-cleaned dataset to teach it the standard Instruction: and Response: conversational format.

⚙️ Recommended Generation Settings

Because this is a compact 350M parameter model, standard generation settings may result in looping or wild hallucinations. For the absolute best outputs, use the following configuration:

• Temperature: 0.5
• Top-K: 50
• Repetition Penalty: 1.3
• Max Length: 400-500
• one can use generation_config.json file in repo

⚠️ Limitations & Bias

• Hallucinations: As a 350M parameter model, Sutra does not have the physical parameter count to act as a factual encyclopedia. It will confidently hallucinate historical dates, math solutions, and trivia.
• Coding: While it understands Python syntax and will output beautifully formatted code blocks (thanks to StarCoder), complex logical scripts may fail.
• Best Use Case: Sutra excels at structural formatting, grammar, summarizing provided context, generate short stories, and acting as a lightweight, lightning-fast local testing model.