pallaslm-110m-sft

🚀 PallasLM 110M SFT (Gated MLA + Sparse MoE)

PallasLM 110M SFT is a high-efficiency Language Model trained on ClimbMix and fine-tuned on SmolTalk, MMLU, and GSM8K. It features state-of-the-art DeepSeek-V3 architectural innovations:

• Gated Multi-Head Latent Attention (Gated MLA)
• Fine-Grained Sparse Mixture-of-Experts (MoE) (8 experts, Top-2 routing)
• Fused Muon + AdamW Optimization

---

📊 Model Architecture Specification

---

📈 Benchmark Evaluation & Performance

Zero-Shot Chat Evaluation (chat_eval.py)

Across 17,590 test questions, PallasLM SFT outperforms random chance baseline across all tasks:

Training & Fine-Tuning Metrics

• Final SFT Training Step Loss: 1.2894 (peak low: 1.2567)
• Minimum Validation BPB: 0.4308 (Bits-Per-Byte)
• Sustained Training Speed: 136,500 tokens/sec on NVIDIA A100

High-Throughput Inference (NVIDIA A100-SXM4-40GB)

---

💻 How to Load Model Weights in PyTorch

import torch

# 1. Load model weights
state_dict = torch.load("model.pt", map_location="cpu")

# State dict keys format:
# - "wte.weight": (32768, 384)
# - "blocks.0..13.mixer...": Gated MLA attention parameters
# - "blocks.4..13.ffn.router.weight": MoE router (8, 384)
# - "blocks.4..13.ffn.experts.0..7...": MoE expert weights

---

💬 Prompt Template Format (ChatML)

During fine-tuning, the model was trained on ChatML conversation tags:

<|bos|><|user_start|>What is the capital of France?<|user_end|><|assistant_start|>

Special Token IDs:

• <|bos|>: Beginning of sequence
• <|user_start|>: Start of user turn
• <|user_end|>: End of user turn
• <|assistant_start|>: Start of assistant turn
• <|assistant_end|>: End of assistant turn (EOS)

---

📜 Training & Fine-Tuning Setup

• Pre-Training: Pre-trained on ClimbMix dataset using the fused Muon + AdamW optimizer (Muon on 2D weight matrices, AdamW on embeddings and LayerNorms).
• Supervised Fine-Tuning (SFT): Fine-tuned on SmolTalk (460K rows), MMLU (auxiliary train), and GSM8K (math reasoning) for 7,049 steps.

---

📄 License

This model repository contains model weight parameters released under the MIT License.