openPangu-2.0-Flash-exl3-4.0bpw

<div align="center">

openPangu · 2.0 · Flash

_{<code>EXL3</code>  ·  <b>4.0 bpw</b>  ·  51.0 GB  ·  Mixture‑of‑Experts  ·  46 layers × 256 experts}

<br/>

</div>

---

[!NOTE] An ExLlamaV3 build of openpangu/openPangu-2.0-Flash at 4.0 bits per weight. See Quants for sibling repos at other bit‑widths or browse the collection.

Quants

<div align="center">

_{KL ÷ fp16: mean KL-divergence from the fp16 source over wikitext rows — lower is closer to the original.}

</div>

Inference

<table> <thead> <tr> <th align="left" width="32%">Loader</th> <th align="left">Use it for</th> </tr> </thead> <tbody> <tr> <td><a href="https://github.com/theroyallab/tabbyAPI"><b>TabbyAPI</b></a></td> <td>OpenAI&#8209;compatible HTTP server. Drop&#8209;in for OpenAI clients.</td> </tr> <tr> <td><a href="https://github.com/oobabooga/text-generation-webui"><b>text&#8209;generation&#8209;webui</b></a></td> <td>Local chat UI. Pick the <i>ExLlamaV3</i> loader from the model dropdown.</td> </tr> <tr> <td><a href="https://github.com/turboderp-org/exllamav3"><b>ExLlamaV3</b></a></td> <td>Direct Python API for embedding the model in your own code or pipeline.</td> </tr> </tbody> </table>

Download

pip install -U huggingface_hub

hf download \
  blockblockblock/openPangu-2.0-Flash-exl3-4.0bpw \
  --local-dir ./openPangu-2.0-Flash-exl3-4.0bpw

<details> <summary><b>Quantization recipe</b> &nbsp;_{(advanced, embedded in <code>quantization_config.json</code>)}</summary> <br/>

Loaded automatically by every ExLlamaV3 loader; reproduced here for searchability.

</details>

License & use

[!IMPORTANT] Use and license follow the base model. Quantization adds no additional restrictions. Refer to the upstream repository for terms, citation, and safety documentation.

---

<div align="center"> _{<i>Quantized with <a href="https://github.com/Honkware/blockquant"><b>BlockQuant</b></a> &nbsp;·&nbsp; convention&nbsp;<code>{org}/{model}-exl3-{bpw}bpw</code></i>} </div>

Requirements

First EXL3 of this architecture. Loading needs exllamav3 with OpenPanguV2 support from Honkware/exllamav3@openpangu; stock exllamav3 releases will not load it yet. mHC, MoME conv, attention sink and DSA indexer tensors ship unquantized.

Multi-token prediction

All three MTP draft layers (46-48) are included and quantized. The per-depth heads and embeddings are tied to the trunk in the base checkpoint, so they are borrowed at load rather than duplicated. Teacher-forced acceptance against the quantized trunk: 92.5% / 89.2% / 88.6% by depth (bf16 reference: 89.2% / 88.6% / 85.9%).

Scope

DSA layers run dense in this build. Batch forward and the MTP chain are validated against the bf16 reference. Cached (paged) generation and per-depth MTP draft dispatch are available on the pangu-paged branch: single-stream chat runs at about 14 tok/s on an H100 and up to 32 tok/s with 3-token MTP drafting (86-96 percent depth-0 acceptance). Batch size 1 only for now; concurrent-job decoding has a known issue under investigation. Quality vs the bf16 original on natural text: mean KL 0.0817, top-1 agreement 92.9%.

License

Use and license follow the base model (LICENSE included in this repo, attribution required). Powered by openPangu.