Ling-3.0-tiny-oQ8e

<p align="center"> <img src="https://uraionlabs.com/public/icons/icon-192.png" alt="Uraion Labs" width="56"> </p> <p align="center"> <strong>Uraion Labs</strong><br> _{High-performance MLX model optimizations for Apple Silicon.} </p>

---

Ling-3.0-tiny — oQ8e (MLX)

<p align="center"> <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny"><img src="https://img.shields.io/badge/Base%20Model-Ling--3.0--tiny-blue?style=for-the-badge&logo=huggingface" alt="Base Model"></a> <a href="https://github.com/ml-explore/mlx"><img src="https://img.shields.io/badge/Framework-MLX%20%2F%20Apple%20Silicon-orange?style=for-the-badge&logo=apple" alt="MLX"></a> <a href="https://huggingface.co/UraionLabs/Ling-3.0-tiny-oQ8e"><img src="https://img.shields.io/badge/Precision-oQ8e%20(8--bit)-success?style=for-the-badge" alt="oQ8e 8-bit"></a> <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny"><img src="https://img.shields.io/badge/Active%20Params-1.3B%20%2F%207.9B-purple?style=for-the-badge" alt="Active Params"></a> <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-green?style=for-the-badge" alt="License MIT"></a> </p>

Ling-3.0-tiny-oQ8e is an 8-bit outlier-aware quantized release of inclusionAI/Ling-3.0-tiny, optimized specifically for Apple Silicon Macs (M1/M2/M3/M4) via MLX and oMLX.

⚡ 8B-class MoE reasoning at 1.3B active parameter speeds (~86–105+ tokens/s) in only ~8.4 GB RAM.

---

🌟 Full Credits & Upstream Attribution

This quantized repository is built upon the breakthrough work of the Ant Ling Team at Ant Group (@inclusionAI).

All credit for the novel hybrid-linear MoE architecture, pre-training corpus, reasoning distillation, thinking mode alignment, and agentic capabilities belongs to the Ant Ling team:

• 🏛️ Original Model Creator: Ant Ling Team (@inclusionAI)
• 📦 Official Base Model: inclusionAI/Ling-3.0-tiny
• 🤖 ModelScope: inclusionAI on ModelScope
• 🐙 OpenRouter API: Ling-3.0-tiny on OpenRouter
• 📖 SGLang Cookbook: Official Ling-3.0-tiny Launch Recipes

Please show your support by starring and checking out the official base model at inclusionAI/Ling-3.0-tiny!

---

⚡ Why Ling-3.0-tiny-oQ8e for Apple Silicon?

Ling-3.0-tiny is one of the most efficient open reasoning models ever created. It couples a hybrid linear-attention architecture (3:1 KDA-MLA) with a sparse 128-expert MoE FFN where only 8 routed experts + 1 shared expert activate per token (only 1.3B active parameters out of 7.9B total).

Highlights of the oQ8e MLX Quantization:

• 🚀 Blazing Fast on Mac: Delivers 86–105+ tokens/second on Apple Silicon (tested on M4 Pro and Apple M-series chips).
• 🧠 Zero Degradation with oQe Quantization: Quantized using oMLX oQe (Outlier-aware Quantization with importance matrix calibration) across all 8,928 MoE expert matrices (oqe_code_multilingual calibration set), preserving complex mathematical, coding, and multi-step reasoning abilities.
• 💾 Low Memory Footprint: Weighs only ~8.39 GB, running comfortably on 16 GB, 24 GB, 36 GB, 48 GB, 64 GB, and 128 GB Unified Memory Macs with headroom for your operating system and IDE.
• 🎯 Native Hybrid Reasoning: Switch seamlessly between instantaneous direct answers and deep <think> chain-of-thought exploration.
• 📚 131K–256K Context: Built for long-context document analysis, multi-file code review, and sustained agentic workflows.

---

🚀 Quickstart: Run in Seconds on Apple Silicon

Option 1: CLI with mlx-lm

Install mlx-lm:

pip install --upgrade mlx-lm

Run immediate text generation:

python -m mlx_lm.generate \
  --model UraionLabs/Ling-3.0-tiny-oQ8e \
  --prompt "<role>HUMAN</role>Explain how quantum computing works in simple terms.<|role_end|><role>ASSISTANT</role>\n<think>" \
  --max-tokens 1024 \
  --temp 1.0 \
  --top-p 0.95

---

Option 2: Python API (mlx_lm)

from mlx_lm import load, generate

# Load model and tokenizer directly from Hugging Face
model, tokenizer = load("UraionLabs/Ling-3.0-tiny-oQ8e")

# Format prompt with Ling's native chat template
prompt = (
    "<role>SYSTEM</role>detailed thinking on<|role_end|>"
    "<role>HUMAN</role>Write a Python script to calculate Fibonacci numbers using dynamic programming.<|role_end|>"
    "<role>ASSISTANT</role>\n<think>"
)

# Generate with recommended sampling parameters
response = generate(
    model=model,
    tokenizer=tokenizer,
    prompt=prompt,
    max_tokens=1024,
    temp=1.0,
    top_p=0.95,
    verbose=True
)

print(response)

---

Option 3: Local OpenAI-Compatible API Server

With omlx (High Performance Apple Silicon Server):

omlx serve UraionLabs/Ling-3.0-tiny-oQ8e --port 8000

With standard mlx-lm:

python -m mlx_lm.server --model UraionLabs/Ling-3.0-tiny-oQ8e --port 8000

Query via curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "UraionLabs/Ling-3.0-tiny-oQ8e",
    "messages": [
      {"role": "user", "content": "What are 3 practical use cases for sparse MoE models?"}
    ],
    "temperature": 1.0,
    "top_p": 0.95
  }'

Query with Python openai client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="UraionLabs/Ling-3.0-tiny-oQ8e",
    messages=[
        {"role": "system", "content": "detailed thinking on"},
        {"role": "user", "content": "Compare KDA attention with traditional multi-head attention."}
    ],
    temperature=1.0,
    top_p=0.95,
)

print(response.choices[0].message.content)

---

🧠 Thinking Mode & Prompt Formatting

Ling-3.0-tiny supports native hybrid reasoning. You can control thinking mode directly inside the system prompt:

• Thinking Mode ON (Deep Reasoning):

  <role>SYSTEM</role>detailed thinking on<|role_end|><role>HUMAN</role>{prompt}<|role_end|><role>ASSISTANT</role>
  <think>
  

• Thinking Mode OFF (Fast Response):

  <role>SYSTEM</role>detailed thinking off<|role_end|><role>HUMAN</role>{prompt}<|role_end|><role>ASSISTANT</role>
  <think></think>
  

Recommended Sampling Parameters:

• Temperature: 1.0
• Top-P: 0.95
• Top-K: 20

---

💻 Hardware Requirements & Apple Silicon Compatibility

Thanks to the ultra-efficient 1.3B active parameter footprint and 8-bit oQe quantization, Ling-3.0-tiny-oQ8e runs smoothly on virtually any Apple Silicon Mac:

Requires macOS Sonoma or Sequoia with MLX installed.

---

📊 Quantization & Model Specifications

---

🔬 Benchmark Performance (Base Ling-3.0-tiny)

Evaluated by the Ant Ling team and Artificial Analysis:

• Artificial Analysis Intelligence Index v4.1.1: 25
• Artificial Analysis Agentic Index: 16
• Inference Speed: Reaches over 160 tokens/s on high-end hardware, with ~86–90 tokens/s measured locally on an M4 Pro MacBook.
• Terminal-Bench 2.1: Strong agentic coding performance under the Artificial Analysis protocol.

---

📜 Citation & Credits

If you use this model in your research or applications, please cite both the original Ant Ling Team and the MLX quantization:

@misc{antling2025ling3tiny,
  title={Ling-3.0-tiny: Efficient Hybrid-Linear Attention Mixture-of-Experts Architecture},
  author={Ant Ling Team},
  organization={Ant Group, inclusionAI},
  year={2025},
  url={https://huggingface.co/inclusionAI/Ling-3.0-tiny}
}

@misc{uraionlabs2025ling3tinyoq8e,
  title={Ling-3.0-tiny-oQ8e: High-Performance 8-Bit MLX Quantization for Apple Silicon},
  author={Uraion Labs},
  year={2025},
  url={https://huggingface.co/UraionLabs/Ling-3.0-tiny-oQ8e}
}

---

<p align="center"> _{Optimized with ❤️ for the Apple Silicon MLX community by <a href="https://huggingface.co/UraionLabs">Uraion Labs</a>.} </p>