Instella-ToolCall-16B-A3B-NVFP4

Instella with Tool Calling — NVFP4

NVFP4 build of our tool-calling fine-tune of amd/Instella-MoE-16B-A3B-Think — AMD's fully-open MoE (16B total / ~2.8B active), taught to call functions.

GGUF / ROCmFP4 sibling (recommended for actual serving today): Instella-ToolCall-16B-A3B-Q4_0_ROCMFP4_STRIX-GGUF

Verified, not assumed

Greedy decode, full token budget, answer taken after </think>:

4/4. The middle two are the interesting ones — they're where quantization damage or a mis-wired architecture surfaces first.

⛔ Two things that will bite you

1. The MoE router must never be quantized. mlp.gate selects which experts fire; 4-bit noise there reroutes tokens to the wrong experts and the damage reads as "the model got dumber" rather than as an obvious failure. It is excluded here, along with lm_head.

2. Do NOT let an engine serve this as DeepSeek-V3. Instella's config declares model_type: deepseek_v3, and vLLM does implement DeepseekV3ForCausalLM while not implementing InstellaMoEForCausalLM. Left to auto-select, it will pick the DeepSeek kernel, skip FarSkip-Collective's dual residual streams, load cleanly and emit fluent-but-wrong text. Force the transformers implementation, and gate on correctness, not on "it loaded".

⚠️ Serving status — read this before downloading

There is currently no inference engine with native instella NVFP4 kernels. Loading via 🤗 transformers works and is verified correct, but compressed-tensors decompresses to bf16 at load, so you save disk, not runtime memory. This artefact is published as a correct, verified NVFP4 checkpoint for anyone adding Instella support to vLLM/SGLang — not as a faster-inference build today.

For running it now, use the ROCmFP4 GGUF sibling above.

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
import torch
M = "kingjones777/Instella-ToolCall-16B-A3B-NVFP4"
cfg = AutoConfig.from_pretrained(M, trust_remote_code=True)
if not hasattr(cfg, "num_local_experts"):          # AMD uses DeepSeek naming
    cfg.num_local_experts = cfg.n_routed_experts
model = AutoModelForCausalLM.from_pretrained(M, config=cfg, trust_remote_code=True,
                                             dtype=torch.bfloat16, device_map="cuda:0")

⚠️ Requires transformers==4.57.1 (4.x line). transformers 5.x breaks AMD's remote code (create_causal_mask() got an unexpected keyword 'input_embeds') and mis-reports every expert as missing.

The fine-tune

LoRA r=64 on the Gated-MLA projections (25.7M params, 0.16%), 2 epochs on 13,171 Apache-2.0 function-calling examples, loss on assistant turns only. Decision accuracy 68.0% → 97.5%; unterminated <think> 121/200 → 0. Full method, evaluation and known limitations are documented on the GGUF card.

Base weights remain under AMD's ResearchRAIL licence — research use only.

<!-- PEER-TABLE:START -->

Other public builds of this model

Compiled from Hugging Face repository metadata — file sizes, shipped files, quant variant as named by each repo. No third-party build was run or benchmarked here, so this table makes no speed or quality claim about any of them. It is here so you can see the size and format options at a glance and pick what fits your hardware.

Base model: amd/Instella-MoE-16B-A3B-Think. Generated from Hub metadata; download counts move over time.

<!-- PEER-TABLE:END --> <!-- CREDITS:START -->

Acknowledgements

Base model authors — see base_model in the metadata above; all model weights, licences and capabilities are theirs. This repository contributes quantisation and measurement only.

llama.cpp / vLLM and NVIDIA's NVFP4 tooling — the formats and runtimes these builds target.

<!-- CREDITS:END -->