Ling-3.0-tiny-GGUF

Ling-3.0-tiny GGUF

Unofficial GGUF conversion and importance-matrix quantizations of inclusionAI/Ling-3.0-tiny, created from immutable source revision a2ee06c0. No fine-tuning, merging, or other parameter training was performed. The original model documentation, intended use, benchmark claims, and limitations remain authoritative.

Experimental runtime requirement

As of 2026-08-11, BailingMoE3 support remains unmerged in upstream llama.cpp. These files were converted and validated with PR #26608 at exact commit d8d8625. This includes the Q-LoRA path required by Ling-3.0-tiny (q_lora_rank=256) from 517b4675 and the pinned multi-argument tool-parser fix 0266ebca. Stock or older llama.cpp binaries and other GGUF runtimes may reject this architecture or produce incorrect output until they incorporate equivalent support.

Preserved model facts

• BailingMoeV3 hybrid KDA/MLA sparse MoE, 526 GGUF tensors
• 7,893,392,800 parameters total; approximately 1.3B active per token
• 24 layers; 128 routed experts, 8 selected per token, plus 1 shared expert
• Q-LoRA rank 256 and KV-LoRA rank 512
• Native configured context: 131,072 tokens
• Embedded tokenizer and source chat template
• No NEXTN/MTP layers (num_nextn_predict_layers=0)

The source identifies itself as Transformers model_type=bailing_hybrid with BailingMoeV3ForCausalLM; the pinned converter intentionally maps that model to GGUF general.architecture=bailingmoe3. This is not a model-family mismatch.

The original card's 256K command uses an external YaRN/runtime override. This release preserves the checkpoint's native 131,072-token configuration and does not claim validated 256K operation. Do not enable MTP speculative decoding for this Tiny checkpoint.

Files and recommendations

If memory permits, prefer Q6_K or Q8_0 for fidelity. Q5_K_M is the quality-oriented general recommendation; Q4_K_M is the lower-memory default. IQ3_M and IQ2_M are specialized memory-constrained choices; the measured loss at IQ2_M is large enough that it should not be a default. File size is not total runtime memory: context length, state/KV caches, backend, and GPU offload add overhead. IQ backend support varies, so use the pinned runtime until equivalent BailingMoE3 support lands elsewhere.

Checksums are in SHA256SUMS.

Download and run

hf download Mike0021/Ling-3.0-tiny-GGUF \
  --include "Ling-3.0-tiny-Q5_K_M.gguf" \
  --local-dir ./models

Build the tested unmerged runtime (review the PR before running it):

git clone --filter=blob:none https://github.com/ggml-org/llama.cpp.git
git -C llama.cpp fetch origin refs/pull/26608/head:pr-26608
git -C llama.cpp checkout d8d862521e9ad842f2b47f3b392b039317782aa0
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
cmake --build llama.cpp/build --config Release --parallel

For a CPU-only build, omit -DGGML_CUDA=ON. This server example deliberately starts at 8K context to keep memory moderate:

./llama.cpp/build/bin/llama-server \
  -m ./models/Ling-3.0-tiny-Q5_K_M.gguf \
  --alias ling-3.0-tiny --host 127.0.0.1 --port 8080 \
  --jinja -c 8192 -ngl 999

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ling-3.0-tiny",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "stream": false
  }'

These sampling settings follow the original model's recommendations. Thinking is enabled by the embedded source chat template by default.

To disable thinking in the pinned server, pass "chat_template_kwargs":{"enable_thinking":false} in the request. Keep --jinja enabled so the embedded template is applied.

The pinned runtime logs special_eos_id is not in special_eog_ids while loading this tokenizer. The raw arithmetic reference stopped on token 156895 in Transformers, and Q4_K_M server stop behavior was tested as described below, but the warning is preserved here because it has not yet been resolved upstream.

Conversion provenance

Importance-matrix calibration

Importance-aware files used two complementary, pinned calibration sources. The primary corpus was lemon07r/bartowski-imatrix-v5-semantic at revision a306f203ee4323e0afe846ae02c2daafe17384d9. Its 2,075 semantic samples span 13 languages and include code, math, science, dialogue, and Q&A, which is substantially broader than English-only WikiText calibration.

An additive second pass used combined_all_micro.parquet from eaddario/imatrix-calibration at revision e87ed55dcba9d9c3a3e41539f3e728e981b1daa4. This MIT-licensed mixture adds multilingual text plus tool-use, math, and code prompts. It was added because the first pass left one routed expert unobserved in one layer; the release gate requires every routed-expert slot to have a nonzero count.

• Input: bartowski-imatrix-v5-semantic.txt
• SHA-256: ff879b5a748f822ef539e43c596a3f44ab922f0295ee209d4220d9f86e86a063
• 1,496,006 bytes; 6,318 serialized lines
• Supplement parquet SHA-256: 94389921e1f67b180a99de28c3090b41ce6f1960eb13abad21b7eba7cbe11b26
• Extracted supplement SHA-256: fdb2d41abf04a2fb207502741a561a5a9ab385eb0c44a450eae676c410955946 (1,008,653 bytes; 3,130 serialized lines)
• Context / batch / ubatch: 4096 / 4096 / 512
• Complete 4,096-token chunks processed: 162 (663,552 tokens); 5,338 trailing tokens excluded
• Matrix entries: 332
• Per-expert count values: 8,832
• Routed-expert slots with zero observations: 0

The matrix is the modern GGUF imatrix format. It contains 69 expert-count vectors of length 128 (8,832 layer/tensor expert slots); “zero” is measured over those slots, not over 128 globally unique expert IDs. Output-tensor statistics were intentionally not collected: the pinned llama.cpp imatrix documentation says it is typically better not to use importance statistics when quantizing output.weight, and therefore defaults --process-output to false. Observed per-slot counts ranged from 16 to 326,023 (median 33,514); a distribution summary and the lowest-count slots are recorded in validation/imatrix.json.

The final matrix SHA-256 is e8b15d131f9ce294f922c5c387f7a69829c12100d6a35bb1635a2b859083c3f0. llama-quantize embeds only one quantize.imatrix.dataset scalar, so the importance-aware model files name the primary corpus even though the final matrix contains both ordered passes. The manifest is the authoritative record of the two-source lineage. It also records the absolute paths embedded by the quantizer; changing those paths can preserve tensor values while changing the GGUF file hash.

The corpus was used only to collect activation statistics. It was not used to train or fine-tune the model and is not an evaluation set.

Held-out validation

Validation used the separate WikiText-2 test file from ggml-org/ci@927b3642933080f1b0e811e2f916e14c292992f9; this file was not used for imatrix collection. Content-level uniqueness from all calibration material or from the model's original pretraining data is not asserted. The extracted wiki.test.raw SHA-256 is 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08. PPL and BF16-relative KLD used 32 fixed sequential chunks at context/batch/ubatch 512, scoring 8,160 held-out tokens. Exact commands are in REPRODUCE.md, and machine-readable results are under validation/.

These tests measure conversion and quantization behavior, not general model capability or safety. Results are comparable only under the documented tokenizer, context, chunk, and pinned-runtime settings. The stored BF16 reference has PPL 11.868127 ± 0.412222. BF16 self-comparison establishes the uint16 stored-log-probability/backend resolution; mean KLD rounded to 0.000000 nats in this run. Small negative ΔPPL values, such as Q8_0, are within sampling uncertainty and do not mean the quant is better than BF16.

“Loads” means the pinned runtime completed its tensor integrity/load check and a graph evaluation. “Greedy raw vs HF BF16” compares a deterministic 12-token continuation against a separately generated Transformers BF16 reference. The validator binds both runtimes to the exact same full prompt; all ten artifacts matched this one shallow case exactly. This is a conversion smoke test, not a claim that quantized logits or arbitrary generations equal BF16. All six tokenizer test cases, including Chinese, code, whitespace, multilingual text, and special tokens, matched Transformers token IDs exactly.

Q6_K contains six Q8_0 fallbacks because those narrow MLA tensors cannot use the requested block width. The 3-bit and 2-bit files likewise contain exactly six documented MLA fallbacks. Their complete tensor-type inventories are in the structure reports and manifest.

Matrix ablation

A direct Q4_K_M A/B against a temporary no-matrix quant gave mixed evidence. The matrix lowered the mean KLD point estimate from 0.131547 to 0.130069 nats and raised the same-top-token point estimate from 84.596% to 85.221%, while PPL moved from 12.357816 to 12.651529. This is not presented as a universal quality gain; the broader calibration coverage and those KLD/same-top point estimate shifts motivated retaining the matrix build. See kld-Q4_K_M-ab.json.

Fixed multiple-choice collapse screen

The pinned mmlu-validation.bin contains 1,548 four-choice tasks. A fixed seed-1 subset of 500 was used as a regression/collapse check, not as a model capability benchmark. The tool's log says “TruthfulQA,” but the supplied input is the pinned MMLU validation binary (SHA-256 470af3a74eccacfaf6f43b08aabf510f61e6c92fe20d17241ded934151e225fa).

Random chance was 25.0% ± 1.9384%. Q8_0 and Q6_K were not run through this auxiliary screen; their held-out KLD results are the stronger fidelity evidence.

Long-context and server checks

BF16, Q4_K_M, and the most aggressive IQ2_M completed a one-chunk 32,768-token perplexity/prefill evaluation at batch 4,096: respectively 23.3709, 25.7803, and 34.6812 PPL. Other artifacts were validated at context 512. The checkpoint's native 131,072-token limit and the external 256K YaRN configuration were not exercised.

Q4_K_M was also tested through llama-server --jinja. Thinking-disabled and thinking-enabled requests both stopped normally, the latter exposed separate reasoning content, a Chinese prompt returned 巴黎, and a required tool request produced get_weather with both location=Paris and unit=celsius arguments and finish_reason=tool_calls. These server results apply to Q4_K_M; they are not generalized to every quant.

Rejected candidates

Two generated candidates were deliberately not published. IQ4_NL was only 28,606,464 bytes smaller than Q4_K_S while its KLD rose from 0.138631 to 0.149734. MXFP4_MOE passed an exact 69-tensor routed-expert whitelist, but at 4,718,248,800 bytes and 0.267021 KLD it was larger and much less faithful than Q4_K_S. On the tested RTX PRO 4500 Blackwell it improved 512-token prompt throughput by 17.9% but reduced 128-token generation throughput by 8.2%. Full measurements are in rejected-candidates.json.

As a post-hoc independent cross-check, the canonical BF16 and Q8_0 SHA-256 values exactly match bloomer010/Ling-3.0-tiny-GGUF@598201. That repository was not used as a weight source.

Limitations and attribution

• Runtime support is experimental and tied to an unmerged llama.cpp revision.
• Quantization can change factuality, reasoning, tool-call formatting, and multilingual behavior; validate the chosen file on your workload.
• Long contexts add substantial memory and were not exhaustively exercised for every artifact.
• No new safety evaluation was performed. The source model's limitations and acceptable-use considerations still apply.
• This is an unofficial conversion, not endorsed by InclusionAI, Hugging Face, or llama.cpp maintainers.

The source card declares the MIT license. Original authorship belongs to InclusionAI; this repository provides an unofficial format conversion by Mike0021.