Qwen3.5-4B-Fix-GGUF

Qwen3.5-4B-Fix — GGUF

llama.cpp builds of LinaTea/Qwen3.5-4B-Fix, which is an unofficial, experimental derivative of Qwen/Qwen3.5-4B-Base: the language model's dense MLP blocks were upcycled to a 2-expert mixture-of-experts (top-1 routing, 512-wide shared expert), then LoRA SFT and GRPO RL were applied on the text side. The vision tower, tokenizer, chat template and image processor are the base model's, unchanged. Not affiliated with, or endorsed by, the Qwen team.

Files

Architecture metadata: qwen35moe, 32 blocks, hidden 2560, 2 experts / top-1, block_count 32, nextn_predict_layers 0 (the base's MTP head is not part of this checkpoint), context 262144.

Use

Image + text (llama.cpp built with mtmd):

llama-mtmd-cli -m Qwen3.5-4B-Fix-Q4_0.gguf \
               --mmproj Qwen3.5-4B-Fix-mmproj-Q8_0.gguf \
               --image photo.jpg -p "Describe this image." -n 256 --temp 0 -ngl 99 --jinja

Text only:

llama-cli -m Qwen3.5-4B-Fix-Q4_0.gguf -p "1+1=?" -n 128 --temp 0 -ngl 99 --jinja

--jinja makes llama.cpp use the chat template embedded in the GGUF, which is the Qwen3.5 template: it pre-opens <think>, so generation starts inside the thought block and the final answer follows </think>. Give it -n 256 or more, or it will still be reasoning when it hits the limit. This build has no --chat-template-kwargs / --no-think switch.

What was tested

• All four language-model quants plus a projector were run through llama-mtmd-cli against a synthetic image with known content (a red circle, a blue square, the text "HELLO 42"): every configuration described it correctly.
• The F16 build was run on a 250,021-token prompt with a needle at the start: the needle was recalled correctly, prefill 4,222 tok/s on one RTX 5090.
• llama-server serves the F16 build with -c 262144 --jinja and returns OpenAI-style tool_calls for a simple function-calling request.

Notes and limitations

• No FP8 or NVFP4 here. llama.cpp has no FP8 or NVFP4 weight-quantization type — its quantizer rejects FP8, FP8_E4M3, NVFP4 and MXFP4 outright and only accepts MXFP4_MOE, which is the 4-bit-float file above. MXFP4 is not the same format as NVIDIA's NVFP4. FP8/NVFP4 checkpoints (e.g. for vLLM/SGLang) are produced by NVIDIA ModelOpt or llm-compressor into compressed-tensors safetensors, a different toolchain.
• Vision is inherited from the base checkpoint, not trained: image understanding is the base model's quality, and no video or audio inputs were validated.
• The RL stage moved held-out GSM8K/MMLU by less than 1 pp, i.e. within noise.
• The base family is prone to repetition loops under greedy decoding; no QKV row clipping was applied to this checkpoint.

License

Apache-2.0, following Qwen/Qwen3.5-4B-Base. Unofficial community build.