MegaSciMoE-1.2B

🧪 MegaSciMoE‑1.2B

<img src="banner.png" width="800" />

A lightweight open MoE language model focused on science reasoning and instruction-following tasks. Fine-tuned using Unsloth on top of LiquidAI’s LFM2‑1.2B architecture.

---

🔬 Model Overview

• Model Type: Sparse Mixture-of-Experts (MoE)
• Base: LiquidAI LFM2‑1.2B
• Fine-tuned with: Unsloth LoRA
• Focus Areas: Physics, Chemistry, Biology, Scientific QA
• Parameter Efficiency: Only 2 experts active (~250M active params)
• Design Goal: Run efficiently on consumer GPUs and edge devices

---

🧠 Capabilities

• Respond to structured science-related instructions
• Generate concise, factual explanations in STEM topics
• Follow multiturn question-answer patterns
• Summarize technical descriptions in plain language

---

🚀 Example Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("yasserrmd/MegaSciMoE-1.2B", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("yasserrmd/MegaSciMoE-1.2B")

prompt = "What is the difference between an ionic and covalent bond?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📚 Training Data

MegaSciMoE-1.2B was fine-tuned on a blend of:

• Multiturn scientific dialogues
• Instruction-based question answering
• Simulated domain-specific prompts in physics, chemistry, and biology
• Scientific corpus extracted from open datasets

---

📎 Sample Prompts

Q: What is a neutron made of?
A: A neutron is composed of three quarks (two down quarks and one up quark)...

---

Q: Define osmosis in simple terms.
A: Osmosis is the movement of water across a membrane...

---

Q: What is the role of mitochondria?
A: Mitochondria are the powerhouses of the cell...

---

🧪 Benchmark Intent

Planned evaluations on:

• ARC-Challenge
• PubMedQA
• ScienceQA (Note: No math-based datasets included in this build)

---

⚙️ Technical Details

• LoRA fine-tuning with Unsloth on 1.2B MoE model
• Memory-efficient inference (fits on T4, L4, RTX 3060, even CPU)
• GGUF and FP16 variants to be released

---

📄 License

Apache 2.0 — free to use and modify with attribution. Ensure upstream license compatibility before commercial use.

---

🤝 Acknowledgements

• Thanks to Unsloth, LiquidAI, and Hugging Face for enabling this build
• Visuals rendered with scientific prompts and surreal-futuristic style

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>