Ling-3.0-tiny-base-midtrain

inclusionAI/Ling-3.0-tiny-base-midtrain

Ling-3.0-tiny-base-midtrain is a machine learning model from inclusionAI. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.

<p align="center" <img src="https://mdn.alipayobjects.com/huameiqa8qxu/afts/img/A4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100"/ </p <p align="center"🤗 <a href="https://huggingface.co/inclusionAI"Hugging Face</…

Downloads · 30 days

544

23% of all-time downloads

All-time downloads

2.3K

Public

Parameters

8.2B

16.4 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors16.4 GB · 100%

Parameter types

How the weights are stored.

BF168.2B · 100%

Model card

From the Hugging Face model README

<p align="center"> <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100"/> </p> <p align="center">🤗 <a href="https://huggingface.co/inclusionAI">Hugging Face</a>   |   🤖 <a href="https://modelscope.cn/organization/inclusionAI">ModelScope </a>   |   🐙 <a href="https://openrouter.ai/inclusionai/ling-3.0-tiny:free">OpenRouter </a>  </p>

Introduction

We have open-sourced the Ling-3.0 series, our most efficient language foundation model family to date. To support research and community-driven innovation, we are releasing a collection of checkpoints during the training process as following:

<div style="width: 100%; overflow-x: auto;"> <table align="center" style="margin: 0 auto; text-align: center; border-collapse: collapse;" > <thead> <tr> <th align="center">Model</th> <th align="center">Pre-trained</th> <th align="center">Mid-trained</th> <th align="center"> Merged (i.e., <a href="https://arxiv.org/abs/2507.17634">WSM</a>) </th> </tr> </thead> <tbody> <tr> <td align="center">Ling-3.0-tiny</td> <td align="center"> <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny-base-30T"> <strong>Ling-3.0-tiny-base-30T</strong> </a> </td> <td align="center"> <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny-base-midtrain"> <strong>Ling-3.0-tiny-base-midtrain</strong> </a> </td> <td align="center"> <a href="https://huggingface.co/inclusionAI/Ling-3.0-tiny-base"> <strong>Ling-3.0-tiny-base</strong> </a> </td> </tr> <tr> <td align="center">Ling-3.0-flash</td> <td align="center"> <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash-base-30T"> <strong>Ling-3.0-flash-base-30T</strong> </a> </td> <td align="center"> <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash-base-midtrain"> <strong>Ling-3.0-flash-base-midtrain</strong> </a> </td> <td align="center"> <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash-base"> <strong>Ling-3.0-flash-base</strong> </a> </td> </tr> </tbody> </table> </div>

These checkpoints correspond to different stages of the training process:

Pretrained checkpoint have completed large-scale pretraining but have not undergone mid-training, WSM merging (or learning-rate decay), or post-training.
Mid-trained checkpoint have completed mid-training but have not undergone WSM merging (or learning-rate decay) or post-training.
Merged checkpoints have undergone WSM merging (or learning-rate decay) based on the mid-training checkpoints but have not undergone post-training.

These checkpoints are released to support continued pretraining, fine-tuning, and further research. For the post-trained model, please see Ling-3.0-tiny and Ling-3.0-flash.

Model Overview

Key features

Highly sparse (1/64) MoE architecture: 512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token;
Native hybrid linear attention: Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
Warmup-Stable and Merge: We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
Scale Seamlessly: Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.

<div style=" display: flex; align-items: center; justify-content: center; gap: 32px; width: 100%; max-width: 1200px; margin: 0 auto; flex-wrap: wrap; " > <div style="flex: 1 1 420px; max-width: 560px; overflow-x: auto;"> <table style="width: 100%; text-align: center; border-collapse: collapse;"> <thead> <tr> <th align="center">Model Type</th> <th align="center">Base (final checkpoint of mid-training)</th> </tr> </thead> <tbody> <tr><td align="center">Architecture</td><td align="center">Hybrid-linear MoE</td></tr> <tr><td align="center">Parameter Scale</td><td align="center">Totoal 7.9B, Activated 1.3B</td></tr> <tr><td align="center">Transformer Layers</td><td align="center">18 KDA + 6 Gated MLA (3:1)</td></tr> <tr><td align="center">Number of Dense Layers</td><td align="center">1</td></tr> <tr><td align="center">Number of Routed Experts</td><td align="center">128</td></tr> <tr><td align="center">Number of Shared Experts</td><td align="center">1</td></tr> <tr><td align="center">Number of Activated Experts</td><td align="center">8</td></tr> <tr><td align="center">Attention Heads</td><td align="center">16</td></tr> <tr><td align="center">Hidden Size</td><td align="center">1536</td></tr> <tr><td align="center">Expert Intermediate Size</td><td align="center">512</td></tr> <tr><td align="center">Dense Intermediate Size</td><td align="center">4608</td></tr> <tr><td align="center">Vocabulary Size</td><td align="center">157,184</td></tr> </tbody> </table> </div> <div style="flex: 0 1 700px; max-width: 800px;"> <img src="https://intranetproxy.alipay.com/skylark/lark/0/2026/png/62256938/1787120625910-bb0c32bd-7e6b-4354-ac65-fa5b747d8bff.png" width="520" style=" display: block; width: 100%; max-width: 800px; height: auto; margin: 0 auto; border-radius: 10px; " alt="" > </div> </div>

Base Model Evaluation

To systematically assess the capabilities of the base model, we use a self-built comprehensive benchmark suite covering several key domains, including knowledge, coding, mathematics, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e., Ling-3.0-tiny-base, is compared below:

Intended Use

Recommended use cases:

Continued pre-training
Mid-training
Supervised fine-tuning for domain adaptation
Preference optimization and RL post-training Distillation research
Long-context and MoE systems research

Not recommended as-is for:

Direct end-user chat deployment
Safety-critical applications without additional alignment and evaluation
Production use without post-training and task-specific validation

Usage

For fine-tuning examples, please refer to our ling-cookbook.

FAQ

If you have any question, please feel free to add a discussion.

License

This model is released under the MIT License.

Type: bailing_hybrid
License: mit
Created: Aug 11, 2026
Updated: Aug 20, 2026