Qwen3.5-4B-Fix

Qwen3.5-4B-Fix (experimental, unofficial)

An unofficial, experimental derivative of Qwen/Qwen3.5-4B-Base, published as Qwen3.5-4B-Fix. The vision tower, tokenizer, chat template and image processor are carried over from the base unchanged; the language model's dense MLP blocks are replaced by a 2-expert mixture-of-experts, and LoRA SFT plus GRPO RL are then applied on the text side only.

Not affiliated with, or endorsed by, the Qwen team.

What the name refers to: the upcycle from the base had dropped the vision tower entirely (a text-only checkpoint whose template still emitted image placeholders), and this build grafts the tower back so the model is genuinely multimodal again. It does not contain the QKV row clipping used elsewhere in this line to cure the family's repetition loops — see Limitations.

What is inside this checkpoint

Scale: 32 layers, hidden 2560, vocab 248320, 2 experts with top-1 routing, shared_expert_intermediate_size 512, tied embeddings, max_position_embeddings 262144. 6.93B parameters, bfloat16, 13.9 GB in three shards.

Usage

Needs transformers >= 5.16 (the version this was built and tested with) and a torch with the model's custom kernels available.

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "REPO_ID"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16,
                                                    device_map="auto")

image = Image.open("photo.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Describe this image in one sentence."},
]}]

text = processor.apply_chat_template(messages, tokenize=False,
                                     add_generation_prompt=True,
                                     enable_thinking=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Several images in one message work the same way: add more {"type": "image", ...} parts and pass the matching list of PIL images to processor(images=[...]).

Text only, either of these works:

messages = [{"role": "user", "content": "1+1=?"}]

# a) through the processor
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
                                       return_dict=True, return_tensors="pt").to(model.device)

# b) through the tokenizer, then the processor
text = processor.tokenizer.apply_chat_template(messages, tokenize=False,
                                               add_generation_prompt=True)
inputs = processor(text=[text], return_tensors="pt").to(model.device)

Thinking mode

The chat template pre-opens the thought block, so with the default enable_thinking=True the generated text starts inside <think> and carries the closing </think> followed by the answer:

We need answer simple. 2+3=5. Need final concise.
</think>

5

With enable_thinking=False the template emits an empty, already-closed thought block and the model answers directly. The model is bilingual; prompts in English or Chinese both work.

System prompt and identity

The Qwen3.5 chat template carries no default system prompt. With no system message the rendered prompt contains no system block at all, and the template raises System message must be at the beginning. if a system message appears anywhere else. This checkpoint was also fine-tuned without any system message, so it has no baked-in identity: it will repeat whatever identity it is handed.

That shows up in practice. Served behind a chat app that injects its own prompt, the model answers "who are you?" with the app's wording, e.g. "I am MoE-SFT-Think, an enhanced mixture-of-experts model provided by Chatbox…" — that text came from the client, not from the weights.

If you serve this model yourself, send a system prompt. One that matches what this checkpoint actually is:

你是 Qwen3.5-4B 的开源社区微调模型：语言部分是 2 专家、top-1 路由的 MoE，视觉塔取自基座， 基于 Qwen/Qwen3.5-4B-Base 改造并做过 SFT 与 RL，属于非官方构建，与阿里云官方发布无关。 支持中英文和图片输入。回答要准确、简洁；不确定就直接说明，不要编造身份或能力。

English equivalent:

You are an unofficial community fine-tune of Qwen/Qwen3.5-4B-Base: a 2-expert top-1 MoE language model with the base model's vision tower, further SFT'd and RL'd on text. You are not an official Qwen release. Be concise and accurate, and say when you do not know.

With that prompt in place the model answers "who are you?" with the description above instead of inventing an identity.

For reference, the last official Qwen default system prompt was in Qwen2.5: its template injected

You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

whenever no system message was given. Qwen3 and Qwen3.5 dropped it — the Qwen3.5-4B instruct template is byte-identical to the base template used here (same 7756-byte file, sha256 a4aee8af…) and contains no system block and no identity sentence at all. Qwen3.5's own identity therefore lives in the instruction-tuned weights, not in any prompt file; this checkpoint, being a fine-tune of the base, has none of it.

GGUF builds

llama.cpp builds live in the sibling repository LinaTea/Qwen3.5-4B-Fix-GGUF: F16, Q8_0, Q4_0 and MXFP4_MOE of the language model, plus F16 and Q8_0 versions of the vision encoder. That repository has the exact llama-mtmd-cli / llama-cli command lines.

This repository holds the original weights only. They are packed in 18 safetensors shards of at most ~800 MB each (model-00001-of-00018.safetensors … ), which keeps every file small enough to upload and re-download individually. The tensor set, dtypes and layout are otherwise unchanged, and AutoModelForImageTextToText.from_pretrained loads them transparently.

FP8 and NVFP4 are not provided in either repository. llama.cpp has no FP8 or NVFP4 weight-quantization type: its quantizer rejects FP8, FP8_E4M3, NVFP4 and MXFP4 outright and only accepts MXFP4_MOE. FP8/NVFP4 checkpoints (for example Unsloth's Dynamic NVFP4, which runs in vLLM/SGLang on Blackwell) are produced by a different toolchain — NVIDIA ModelOpt / llm-compressor into compressed-tensors safetensors — not by GGUF tooling.

What was tested

• Images with known ground truth (synthetic shapes, colours, printed text) and real documents (a LaTeX paper page): colours, shapes and text came back correctly, and OCR-level details such as a paper's title, author and date were read correctly.
• Multiple images per message, thinking and non-thinking modes, and both chat template routes shown above.
• Identity handling with and without a system prompt: without one the model repeats a client-supplied identity verbatim; with the system prompt above it describes itself correctly.
• Long context: a needle at the start of a 255,010-token prompt was retrieved correctly (1,895 tok/s prefill on one RTX 5090, using a memory-bounded chunked prefill loop). Stock generate() builds the whole prompt's attention in one pass, so practical context length there is set by available VRAM rather than by this model's 262144 limit.
• Served as a local OpenAI-compatible endpoint with streaming.

Limitations

• Vision is inherited, not trained. The vision tower is the base model's; the language model was only ever fine-tuned on text. Image understanding is therefore the base model's quality, about a dozen images were checked, and no visual benchmark suite was run.
• Video and audio were not tested. A video preprocessor config is present and the model class accepts video_grid_thw, but nothing here was validated on video. The audio tokens have no audio tower in this checkpoint.
• The base family is prone to repetition loops under greedy decoding, caused by very large outlier rows in the QKV projections. No QKV row clipping was applied here; the fix used for the smaller checkpoints of this line was to rescale rows whose norm exceeds 3× the median row norm in the QKV projections.
• The RL stage was a wash. On held-out GSM8K and MMLU it moved scores by less than 1 pp, inside the standard error.
• Experimental research artifact; no safety tuning beyond whatever the base carries. Do not expect instruction-following quality comparable to a released chat model.

License and attribution

Apache-2.0, following Qwen/Qwen3.5-4B-Base; the full LICENSE is included in this repository. The weights derive from that base checkpoint: the vision tower and tokenizer are unmodified copies, the language model was structurally modified (dense → 2-expert MoE) and then fine-tuned as described above. This is an unofficial community build.