Instella-ToolCall-16B-A3B-ROCmFP4-STRIX-GGUF

🔧 Runtime: build the ROCmFPX fork below

Stock llama.cpp will not load this file. You need both the instella architecture and the ROCmFP4 tensor types in one tree. Upstream charlie12345/ROCmFPX has the ROCmFP4 types but not instella. Our fork has both:

kingjones30/ROCmFPX — a fork of charlie12345/ROCmFPX, branch main.

git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-quantize -j$(nproc)

Verified 2026-08-27 on gfx1151: clean clone → 0 build errors → llama-server loads a instella ROCmFP4 GGUF from this family and generates coherent text.

Instella with Tool Calling — ROCmFP4 STRIX GGUF — AMD Ryzen AI Max+ 395 / Strix Halo / gfx1151

AMD shipped a reasoning model that cannot call a tool. We taught it how.

This is amd/Instella-MoE-16B-A3B-Think — AMD's fully-open MoE (16B total / ~2.8B active) — fine-tuned for function calling and agentic tool use, then quantized to ROCmFP4 for AMD Strix Halo (gfx1151).

To our knowledge this is the first tool-calling-capable Instella. AMD never trained or evaluated the model for it: the card doesn't mention tools, and the eval suite (WinoGrande, HumanEval+, IFEval, HELMET, RULER) contains no agentic or function-calling benchmark at all.

---

What actually changed

Per capability the tuned model gets 19/19 parallel calls, 8/8 single calls, and makes only 2 false calls in 61 no-call cases.

🔍 The bug was hiding behind a good-looking number

The base model scores 85.2% on "correctly did not call a tool" — which looks like judgement.

It isn't. 52 of those 61 correct-looking rows had an unterminated <think> block — exactly 85.2%. The model wasn't declining. It was thinking forever and never acting. Here it is reasoning perfectly and then doing nothing:

<think> We are given a request to book a flight… the available tools do not include any booking functionality. The only tool we have is generate_random_color, which is unrela…

and on a case where it should have called:

<think> …we have a function search_recipes that takes ingredients… we can use it. However, note that t…

It concludes correctly, then never emits the call. The metric was rewarding a bug.

That single defect wears two faces:

• thinking on → never terminates, never acts
• thinking suppressed (needed to get the format right) → no reasoning phase, so it fires blindly: "What is 2+2?" → get_weather{"city": "New York"}

The tuned model now declines for reasons:

"I'm sorry, but I'm unable to assist with that. My current capabilities are limited to calculating loan payments. I don't have the ability to book flights."

---

How it was trained

LoRA, rank 64, on the Gated-MLA projections — q_proj, o_proj, kv_a_proj_with_mqa, kv_b_proj — 25.7 M trainable params (0.16% of 15.9 B). 2 epochs, lr 1e-4 cosine, bf16, gradient checkpointing, length-grouped batching, max-len 4096. Loss 0.56 → 0.13. ~6 h 40 m on a single NVIDIA GB10.

Data — 13,171 examples, Apache-2.0 only, from glaiveai/glaive-function-calling-v2 and NousResearch/hermes-function-calling-v1. CC-BY-NC sources were deliberately excluded to keep the licence chain clean.

Three decisions did most of the work:

1. Loss on assistant turns only. A {% generation %}-marked training template plus return_assistant_tokens_mask means the model is never trained to reproduce the tool schemas sitting in its own prompt. Verified by decoding the mask: 460 tokens in, 52 supervised.
2. Negative examples are the whole point. 30% of the corpus is tools offered, correct answer is prose. Judgement is a learned behaviour — you cannot prompt it in.
3. Truncation was checked per capability, not globally. At max-len 1536 we were silently dropping 37.5% of parallel-call examples — the scarcest capability in the mix. 4096 keeps 100%.

⚠️ What we tried that made it worse

A second pass on a 50% negative mix improved every held-out metric — decision 97.5% → 98.0%, no_call 96.7% → 98.4% — and regressed on an out-of-distribution probe, newly failing "What is 2+2?", which v1 gets right.

More data of the same shape ≠ better generalisation. It learned Glaive's refusal phrasing harder without learning the principle. We shipped v1. In-distribution eval alone would have shipped the worse model — keep an adversarial probe that disagrees with your training distribution.

Known limitations (measured, not hedged)

On a 12-question adversarial sweep — general-knowledge questions with one irrelevant tool offered — it makes 2 false calls:

• "What is 17*23?" → calls the tool
• "Translate 'hello' to Spanish." → calls the tool

Both resemble tool-able operations, and Glaive is dense with calculate_* / translate_* tools, so it learned "this task type is tool-able" without fully learning "is this tool relevant." Everything else answers correctly: 2+2, 100÷4, √144, leap year, Hamlet, boiling point, photosynthesis, WW2, capital of France, largest planet.

It is also a 2.8 B-active model. It is a fast, decisive tool-caller — not a frontier coder.

---

⛔ Requirements — read before downloading

1. A ROCmFPX runtime. ROCmFP4 uses ggml tensor types 100–106; stock llama.cpp refuses the file:

gguf_init_from_reader: tensor 'output.weight' has invalid ggml type 101. should be in [0, 43)

Build ROCmFPX — both llama-quantize and llama-server.

2. The instella architecture. Not in llama.cpp mainline. Instella declares model_type: deepseek_v3 and reuses DeepSeek MLA shapes, but it is not a drop-in DeepSeek-V3 — FarSkip-Collective needs dual residual streams, and a naive port loads fine and emits fluent-but-wrong text. Plus Gated MLA, rope_interleave, qk_layernorm.

3. instella-tools-tuned.jinja (included) is mandatory for tool calling. Instella inherits DeepSeek's chat template verbatim, which has no tools variable — it can format a tool call from history but can never advertise available tools, so the model never learns what it may call. This is a known, unresolved upstream DeepSeek issue (DeepSeek-V3 #48). Our template keeps Instella's trained turn markers exactly and adds a real <tools> block.

4. --reasoning on is required. --reasoning defaults to auto, which resolves to off for a custom template and will silently disable thinking on a Think model.

Serving

llama-server -m Instella-ToolCall-v1-STRIX-imat.gguf \
  --host 0.0.0.0 --port 8087 \
  -ngl 999 -dio --no-warmup --jinja \
  --chat-template-file instella-tools-tuned.jinja \
  --reasoning on --reasoning-format deepseek \
  -c 32768 --parallel 1 --alias instella-toolcall-v1

Standard OpenAI tools / tool_choice; calls come back in message.tool_calls, reasoning in message.reasoning_content.

Also included

• instella-tools-tuned.jinja — the serving template (required)
• adapter/ — the LoRA adapter (108 MB) if you'd rather merge it yourself
• llamacpp-instella-toolcall.patch — two upstreamable common/chat-diff-analyzer.cpp fixes: registering the legacy-DeepSeek tool markers as preserved tokens (without which llama.cpp hard-400s any request carrying tools), and a </tool_call> stop guard for untuned checkpoints. The first also fixes DeepSeek-R1, R1-Distill and pre-DSML V3.

Credits

AMD for a genuinely open model — weights, data mixtures and training code. Glaive and Nous Research for Apache-2.0 function-calling data. ROCmFPX for the ROCmFP4 quantizer.

Base weights remain under AMD's ResearchRAIL licence — research use only. The LoRA and template are ours; the training data is Apache-2.0.

<!-- PEER-TABLE:START -->

Other public builds of this model

Compiled from Hugging Face repository metadata — file sizes, shipped files, quant variant as named by each repo. No third-party build was run or benchmarked here, so this table makes no speed or quality claim about any of them. It is here so you can see the size and format options at a glance and pick what fits your hardware.

Base model: amd/Instella-MoE-16B-A3B-Think. Generated from Hub metadata; download counts move over time.

<!-- PEER-TABLE:END --> <!-- CREDITS:START -->

Acknowledgements

This build would not exist without the work below. Please star and follow these projects — the quantisation format used here is their engineering, not mine.

ROCmFPX — maintained by charlie12345 / caf The ROCmFP4 / ROCmFPX tensor formats (ggml types 100–106) exist only in this fork. Every ROCmFP4 file in this repository was produced with its llama-quantize, and runs on its runtime. The fork also credits collaborators ciru-ai, Tom Turney, PlunderStruck and Aydan S., and acknowledges AMD for hardware support. Licensed MIT, based on upstream llama.cpp.

llama.cpp — ggml-org and contributors The inference engine, GGUF format and conversion tooling everything here is built on.

AMD ROCm The compute platform these builds target — ROCm 7.2.4 on gfx1151 / Radeon 8060S.

Base model authors — see base_model in the metadata above; all model weights, licences and capabilities are theirs. This repository contributes quantisation and measurement only.

If you use these files, please credit ROCmFPX alongside this repository.

<!-- CREDITS:END -->