qwen3.5-4b-ninfer

Qwen3.5-4B for NInfer

This model card is the version-controlled source for v381654729/qwen3.5-4B-ninfer.

The repository contains Qwen3.5-4B converted to the native NInfer .ninfer artifact format. The artifact is intended only for NInfer; it is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Artifact

The file contains the registered Text, Vision, MTP, proposal-head, tokenizer, chat-template, generation, and media-processor objects required by NInfer.

Verify a downloaded file with:

printf '%s  %s\n' \
  '480d63e61cbc8d5e1375daf9b766a1dab1cd0df7668fb388ea3a45f98d6372cb' \
  'qwen3_5_4B.ninfer' | sha256sum --check

Source model

Qwen3.5-4B is a multimodal dense model with a mixed linear/full-attention text backbone:

Requirements

• NInfer revision, built from source;
• 64-bit Linux;
• Verified on​ NVIDIA GeForce RTX 5090 (laptop, sm_120a). Also possibly expected to work on other GPUs (e.g., RTX 5080, 5060 Ti), but these have not been tested.
• CUDA Toolkit 13.1 or newer.

Download and run

hf download v381654729/qwen3.5-4b-ninfer \
  qwen3_5_4B.ninfer \
  --local-dir models

./build/apps/ninfer models/qwen3_5_4b.ninfer \
  --prompt "What is the capital of France?" \
  --max-context 4096 \
  --max-new 256

Serve it through the OpenAI-compatible HTTP endpoint:

./build/apps/ninfer-serve models/qwen3_5_4b.ninfer \
  --host 0.0.0.0 \
  --port 8000 \
  --vision \
  --max-context 262144 \
  --kv-capacity 262144 \
  --default-max-tokens 16384 \
  --max-concurrency 1 \
  --device-state-slots 4 \
  --host-state-slots 4 \
  --host-kv-mib 8192 \
  --prefill-chunk 4096 \
  --kv-dtype fp8 \
  --spec mtp \
  --lm-head-draft \
  --draft-tokens 3 \
  --top-k 20 \
  --top-p 0.95 \
  --temperature 0.01 \
  --preserve-thinking \
  --model-id Qwen3.5-4B

Supported use

The artifact supports:

• text generation in thinking and non-thinking modes;
• image, multi-image, video, and mixed multimodal messages;
• MTP speculative decoding with draft windows from one to five;
• CUDA Graph decode and compatible-prefix reuse;
• the NInfer CLI;
• OpenAI Chat Completions and Anthropic Messages serving.

License

This NInfer artifact is distributed under the Apache License 2.0. The source Qwen3.5-4B repository is also licensed under Apache-2.0. Users remain responsible for complying with the license and applicable laws.