Qwen3.5-4B-EU-Tool-Q4_K_M-GGUF

BobbyLLM/Qwen3.5-4B-EU-Tool-Q4_K_M-GGUF

Q4_K_M GGUF conversion of birgermoell/Qwen3.5-4B-EU-Tool.

The original conversion was produced with llama.cpp through the GGUF-my-repo Space.

Refer to the original model card for model details, training information, supported languages, and the original tool-use evaluation.

Files

Two Q4_K_M GGUF files are provided.

Recommended

qwen3.5-4b-eu-tool-q4_k_m-no-nextn-test.gguf

This is the same Q4_K_M quantisation as the original conversion. The tensors were not requantised or modified.

Two GGUF metadata values were corrected after the original conversion:

qwen35.block_count: 33 -> 32
qwen35.nextn_predict_layers: 1 -> 0

The original GGUF declared 33 blocks, but its tensor inventory contains only:

blk.0 ... blk.31

There are no blk.32 tensors.

Before the metadata correction, llama.cpp/BeeLlama failed with:

check_tensor_dims: tensor 'blk.32.attn_norm.weight' not found

The corrected GGUF successfully loaded in the tested BeeLlama/llama.cpp build when MTP was disabled.

Original conversion

qwen3.5-4b-eu-tool-q4_k_m.gguf

This is the untouched output produced by GGUF-my-repo.

It is retained for reference and for users who want the original conversion.

On the tested BeeLlama/llama.cpp build, this file failed to load because of the 33-block metadata mismatch described above.

Important: use the supplied chat template

The GGUF-my-repo conversion did not preserve the source model's tokenizer.chat_template.

This matters for tool use.

Without the source template, the tested model could generate normal text but did not emit working llama.cpp/MCP tool calls.

The original SFT template is therefore also included in this repository:

Qwen3.5-4B-EU-Tool-chat_template.jinja

For llama.cpp tool use, enable Jinja and explicitly load this template:

--jinja
--chat-template-file Qwen3.5-4B-EU-Tool-chat_template.jinja

The supplied template comes from the source model:

birgermoell/Qwen3.5-4B-EU-Tool

Its LF-normalised SHA-256 is:

a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715

Important: disable MTP

The recommended patched GGUF does not contain MTP layers.

Use:

--spec-type none

Do not launch it with:

--spec-type draft-mtp

Doing so caused the tested llama.cpp/BeeLlama build to fail with:

context type MTP requested but model doesn't contain MTP layers
failed to create MTP context

Recommended llama.cpp usage

Download these two files:

qwen3.5-4b-eu-tool-q4_k_m-no-nextn-test.gguf
Qwen3.5-4B-EU-Tool-chat_template.jinja

Server

llama-server \
  -m qwen3.5-4b-eu-tool-q4_k_m-no-nextn-test.gguf \
  --jinja \
  --chat-template-file Qwen3.5-4B-EU-Tool-chat_template.jinja \
  --spec-type none

For GPU offload, add an appropriate -ngl value for your hardware.

For example, to request full available layer offload:

llama-server \
  -m qwen3.5-4b-eu-tool-q4_k_m-no-nextn-test.gguf \
  -ngl 99 \
  --jinja \
  --chat-template-file Qwen3.5-4B-EU-Tool-chat_template.jinja \
  --spec-type none

CLI

llama-cli \
  -m qwen3.5-4b-eu-tool-q4_k_m-no-nextn-test.gguf \
  --jinja \
  --chat-template-file Qwen3.5-4B-EU-Tool-chat_template.jinja \
  --spec-type none \
  -p "Hello"

Tool-use note

In local testing, tool calling was effectively blocked when the external source chat template was not supplied.

With the source chat_template.jinja loaded through llama.cpp, valid MCP tool calls worked again.

Successful tested calls included:

Kiwix search
Kiwix article fetch
Exa web search

This proves that the model can emit working tool calls with the correct template.

It does not prove that the model will select the correct tool, evidence source, or multi-step retrieval strategy for every task.

Conversion notes

The recommended GGUF is not a new quantisation.

The Q4_K_M tensor data from the GGUF-my-repo conversion remains unchanged.

Only these GGUF metadata fields were altered:

qwen35.block_count
qwen35.nextn_predict_layers

The source chat template is supplied separately because the conversion omitted the embedded tokenizer.chat_template.

Credits

Original model:

birgermoell/Qwen3.5-4B-EU-Tool

GGUF conversion:

ggml-org/gguf-my-repo

llama.cpp:

ggml-org/llama.cpp