Instella-MoE-16B-A3B-Think-GGUF

Instella-MoE-16B-A3B-Think — High-Quality F16 GGUF

A clean F16 GGUF conversion of AMD's Instella-MoE-16B-A3B-Think, using the model's native instella GGUF architecture.

This repository is intentionally centered on one artifact:

Instella-MoE-16B-A3B-Think-F16.gguf

The F16 file is my Gold Standard Reference for subsequent GGUF quantization, runtime, and ROCmFPX experiments.

Experimental IQ, Q, and ROCmFP4 derivatives may be produced from this file or from the original AMD checkpoint, but they are considered downstream work and are not the reference release represented by this repository.

Strix support will come after I build the ROCmFP4 GGUFs and get all the runtimes sorted out.

---

Current Release

• SHA256: 4aa62f3a14277262bb50d0ca270fc400c74c5eea2054e717534ab5f5b3d2d585 | Property | Value | | :--- | :--- | | Model | Instella-MoE-16B-A3B-Think | | GGUF file | Instella-MoE-16B-A3B-Think-F16.gguf | | GGUF architecture | instella | | File type | F16 | | Total parameters | 16B | | Active parameters/token | ~2.8B | | GGUF tensors | 457 | | Vocabulary | 128,896 | | Context length | 32,768 | | Approx. file size | ~32 GB | | Release status | Gold-master reference |

Status: The F16 GGUF is the stable reference point. Downstream quantization variants are experimental and should be evaluated against this file.

---

Model

This GGUF is converted from the official AMD checkpoint:

amd/Instella-MoE-16B-A3B-Think

AMD describes Instella-MoE as a fully open Mixture-of-Experts language model with approximately 16 billion total parameters and approximately 2.8 billion active parameters per token.

The Think checkpoint is the final RL-refined checkpoint in AMD's released training pipeline.

Instella contains several architectural features that matter directly to GGUF conversion and inference:

• Gated Multi-head Latent Attention (Gated MLA)
• FarSkip-Collective
• 27 decoder layers
• 2048 hidden size
• 16 attention heads
• 1 KV head
• 64 routed experts
• 2 shared experts
• 6 activated experts per token
• 128,896 vocabulary
• 32,768-token context
• YaRN RoPE scaling

AMD's original checkpoint identifies the Transformers architecture as:

InstellaMoEForCausalLM

while the configuration also uses the deepseek_v3 model-family designation.

That combination is important: although Instella contains DeepSeek-V3-style MLA structures, it is not simply a generic DeepSeek-V3/DeepSeek2 checkpoint.

---

Why this GGUF uses instella

Instella superficially resembles DeepSeek-V3 because it uses related MLA and MoE structures.

However, Instella also implements model-specific behavior, including:

• Gated MLA
• FarSkip-Collective
• Instella-specific attention behavior
• Instella-specific tensor handling

For that reason, this GGUF intentionally uses:

general.architecture = instella

rather than:

general.architecture = deepseek2

The goal is to preserve the actual Instella computation graph instead of forcing the checkpoint through a generic architecture description.

The dedicated instella architecture is therefore a fundamental part of this conversion, not merely a metadata preference.

---

F16 GGUF Conversion

The F16 GGUF was rebuilt from the original AMD BF16/Safetensors checkpoint using the kingjones30 ROCmFPX codebase with native Instella GGUF architecture support.

The conversion path is:

AMD Instella BF16 / Safetensors
                │
                ▼
      InstellaMoEForCausalLM
                │
                ▼
   Instella-aware GGUF conversion
                │
                ▼
      GGUF architecture = instella
                │
                ▼
       F16 GGUF gold master

The resulting GGUF contains:

general.architecture = instella
general.file_type    = F16
GGUF tensors         = 457

---

Verified GGUF Metadata

The rebuilt F16 GGUF carries the following Instella-specific metadata:

block_count                  = 27
embedding_length             = 2048
vocab_size                   = 128896

expert_count                 = 64
expert_shared_count          = 2
expert_used_count            = 6
expert_feed_forward_length   = 1408

attention.head_count         = 16
attention.head_count_kv      = 1
attention.kv_lora_rank       = 512
attention.key_length         = 544
attention.value_length       = 512

attention.key_length_mla     = 128
attention.value_length_mla   = 128

context_length               = 32768

rope.dimension_count         = 32
rope.freq_base               = 8000000
rope.scaling.type             = YaRN
rope.scaling.factor           = 40
rope.scaling.original_context_length = 4096

---

Tokenizer Verification

One of the important lessons from the earlier conversion work was that a GGUF can have the right weights and still be wrong at the tokenizer / metadata layer.

The current F16 master was therefore rebuilt and explicitly inspected rather than accepting a successful conversion as sufficient verification.

The verified tokenizer metadata is:

tokenizer.ggml.model       = gpt2
tokenizer.ggml.pre         = joyai-llm

tokenizer.ggml.bos_token_id = 0
tokenizer.ggml.eos_token_id = 1
tokenizer.ggml.padding_token_id = 1

tokenizer.ggml.add_bos_token = True
tokenizer.ggml.add_eos_token = False

The GGUF tokenizer contains:

GGUF tokenizer token count = 128896

with the expected Instella special-token region beginning with:

<｜begin▁of▁sentence｜>
<｜end▁of▁sentence｜>
<｜▁pad▁｜>

and the vocabulary ending with the expected padded-token entries.

The tokenizer is treated as part of the model conversion itself, not as an afterthought.

---

Instella Gated Attention Tensor

Instella's gated attention is part of the actual architecture and must survive conversion correctly.

The F16 GGUF preserves a dedicated attention-gate tensor for every decoder block:

blk.N.attn_gate

This is important because the attention gate is part of the model's computation rather than optional metadata.

A downstream quantization investigation also identified the corresponding source-to-GGUF tensor mapping:

HF:

model.layers.{i}.self_attn.gate_proj.weight

        ↓ conversion

GGUF:

blk.{i}.attn_gate

This mapping is now treated as an explicit validation point when producing downstream quantized derivatives.

In other words, a quantizer can no longer be considered trustworthy merely because it produces a syntactically valid GGUF. The resulting tensor map must also preserve the Instella architecture correctly.

---

Conversion Lessons

This repository exists partly because the first conversion was not the final answer.

The earlier conversion was represented using a generic DeepSeek architecture. That version was not retained as the release master.

The F16 reference was rebuilt from the original AMD BF16 checkpoint with the Instella-aware conversion path.

The resulting release reference is:

Architecture : instella
File type    : F16
Tensors      : 457
Size         : ~32 GB
Vocabulary   : 128896
Context      : 32768

The tokenizer and metadata were also rechecked after the rebuild.

This gives downstream quantization work a known-good starting point rather than attempting to debug model architecture, tokenizer metadata, tensor mapping, and quantization behavior simultaneously.

---

Why F16 Is the Gold Master

The purpose of keeping a full-precision reference is reproducibility.

All downstream quantization work should be evaluated against the F16 master for:

1. Tensor names and tensor count
2. GGUF architecture metadata
3. Tokenizer metadata
4. Expert configuration
5. Attention configuration
6. Special Instella tensors such as blk.N.attn_gate
7. Runtime behavior

The F16 file is therefore the control artifact for future quantization experiments.

A smaller quantized file may be useful for inference, but it should not replace the F16 reference when determining whether a conversion pipeline is correct.

---

Quantization Work

Experimental quantization work is intentionally kept downstream from the F16 master.

Experiments have included standard GGUF formats such as:

Q4_K_S
Q5_K_M
IQ4_XS

as well as AMD-specific ROCmFP4 work using ROCmFPX.

The important distinction is:

                 GOLD MASTER
                     │
                     ▼
        Instella-MoE-16B-A3B-Think-F16.gguf
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
        Q/IQ      ROCmFP4    other tests
        variants   variants

Quantized derivatives should be considered experimental until they have been validated against the F16 reference.

---

Provided Quants

(sorted by size. The IQ-quants are often preferred over similar sized Base/Static quants)

---

ROCmFPX / ROCmFP4

This project also explores AMD-focused low-bit formats using the specialized ROCmFPX runtime.

Stock llama.cpp builds do not necessarily support experimental ROCmFP4 tensor layouts.

For AMD users, the intended environment is a ROCm-compatible GPU together with an appropriate ROCmFPX build targeting the local GPU architecture.

Example target:

gfx1201

The F16 GGUF is deliberately kept separate from these experimental tensor formats so that the base model remains easy to inspect and validate.

---

ROCmFPX Build Example

Clone the specialized ROCmFPX tree:

git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX

Configure for an AMD gfx1201 target:

cmake -B build -S . \
  -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1201 \
  -DGGML_NATIVE=ON \
  -DCMAKE_BUILD_TYPE=Release

Build the server and CLI:

cmake --build build --target llama-server llama-cli -j$(nproc)

Adjust the CMake options as required by the exact ROCmFPX revision and local ROCm installation.

---

Example ROCmFPX Runtime

A typical ROCmFPX test invocation is:

cd ~/ROCmFPX

env -u LD_LIBRARY_PATH \
  HIP_VISIBLE_DEVICES=0 \
  ./build/bin/llama-cli \
  -m Instella-MoE-16B-A3B-Think-F16.gguf \
  -ngl 999 \
  -c 2048 \
  --temp 0.8 \
  -n 32 \
  -p "Give me a short technical explanation of Instella's gated attention."

For a quantized ROCmFP4 derivative, the same runtime must be used with the corresponding ROCmFPX-compatible GGUF.

Also see = https://github.com/csabakecskemeti/llama.cpp/tree/instella-moe for another instella compatible server.

---

Hardware / Development Environment

The reference conversion and validation work was performed in an AMD-oriented development environment.

Representative test hardware:

CPU   : AMD Ryzen 7 9800X3D
RAM   : 32 GB DDR5-6000
GPU   : AMD Radeon RX 9070 XT
VRAM  : 16 GB
Target: gfx1201
ROCm  : 7.2.x

The full F16 GGUF is approximately 32 GB and therefore does not fit entirely in 16 GB of VRAM.

The F16 conversion itself was performed CPU-side from the original BF16/Safetensors checkpoint.

The intended purpose of the F16 file is therefore reference validation and a source for later quantization rather than direct full-residency inference on a 16 GB GPU.

---

Validation Philosophy

A successful GGUF conversion is not defined solely by:

"the file exists"

Validation should cover the whole chain:

Original checkpoint
        │
        ▼
Architecture mapping
        │
        ▼
Tensor names / tensor count
        │
        ▼
GGUF metadata
        │
        ▼
Tokenizer
        │
        ▼
Special architecture tensors
        │
        ▼
Runtime loading
        │
        ▼
Actual generation

For Instella, particular attention should be paid to:

general.architecture
tokenizer.ggml.*
attention.*
expert.*
rope.*
blk.N.attn_gate

This is especially important when evaluating downstream quantizers, because a conversion can appear healthy while silently altering or mishandling an architecture-specific tensor.

---

Repository Scope

This repository is intentionally maintained as the clean F16 reference for Instella GGUF work.

The current release is:

Instella-MoE-16B-A3B-Think-F16.gguf

Experimental quantization artifacts may be produced during development, but the F16 file remains the authoritative reference.

That separation makes it possible to answer a useful question when something breaks:

Is the problem in the original model conversion, or in the downstream quantization/runtime?

The F16 master provides the control needed to answer that.

---

Original Model

Original AMD model:

amd/Instella-MoE-16B-A3B-Think

Please consult the original AMD model card and license before using or redistributing the model.

AMD's original release includes the model weights, tokenizer, configuration, inference code, and training-stage information.

---

License

The upstream AMD checkpoint is released under the Research Rail License.

Please consult the original AMD repository for the complete and current license terms before redistribution or commercial use:

https://huggingface.co/amd/Instella-MoE-16B-A3B-Think/blob/main/LICENSE

This GGUF conversion does not supersede or replace the upstream model's license.

---

Citation

If you use Instella-MoE, please cite the AMD Instella-MoE technical report:

@article{instella-moe,
  title={Instella-MoE Technical Report},
  author={Liu, Jiang and Ranjan, Sudhanshu and Mishra, Prakamya and Dukler, Yonatan and Ramesh, Gowtham and Wu, Jialian and Sun, Ximeng and Xie, Wen and Hou, Chaojun and Appia, Vikram and Gu, Zhenyu and Liu, Zicheng and Barsoum, Emad},
  journal={arXiv preprint arXiv:2609.00791},
  year={2026}
}

For the architectural FarSkip-Collective work:

@inproceedings{
  dukler2026farskipcollective,
  title={FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models},
  author={Yonatan Dukler and Guihong Li and Deval Shah and Jiang Liu and Vikram Appia and Emad Barsoum},
  booktitle={Ninth Conference on Machine Learning and Systems},
  year={2026},
  url={https://openreview.net/forum?id=ruOpvLzsGV}
}

---

Acknowledgements

This work builds on the original AMD Instella-MoE release and the community GGUF / ROCmFPX ecosystem.

Custom conversion and runtime work involved ROCmFPX implementation forks associated with:

charlie12345/ROCmFPX
kingjones30/ROCmFPX

Thanks to everyone contributing to AMD GPU inference, GGUF conversion, quantization research, OpenAI, and the broader llama.cpp ecosystem.

---

Experimental Status

Instella-MoE and the surrounding GGUF / ROCmFPX work are still evolving.

Quantized derivatives may contain runtime-, architecture-, or quantization-specific issues even when the underlying F16 master is correct.

For that reason:

Use the F16 GGUF as the reference when evaluating experimental derivatives.

A derivative should not be considered equivalent to the F16 master merely because it loads successfully.

---

Final Reference

The intended hierarchy for this repository is:

AMD official BF16 / Safetensors
              │
              ▼
   Instella-aware F16 GGUF
              │
              ▼
   GOLD MASTER / CONTROL
              │
      ┌───────┼────────┐
      ▼       ▼        ▼
     Q/IQ   ROCmFP4   Other
   variants variants  experiments

The central artifact remains:

Instella-MoE-16B-A3B-Think-F16.gguf

with:

Architecture : instella
File type    : F16
Tensors      : 457
Vocabulary   : 128896
Context      : 32768
Size         : ~32 GB

This F16 GGUF is the gold-standard reference for the project.