Karti-Small-VL-4B-NVFP4

Karti-Small-VL-4B · v1 · NVFP4

2.5× the throughput, 40% of the size — and it can still see.

NVFP4 build of KartiOS/Karti-Small-VL-4B v1, a 4B vision + tool-calling model for local agents.

Speed — NVIDIA DGX Spark (GB10)

vLLM 0.27.1, 32k context, FP8 KV cache, temperature 0, single stream, best of 3.

It also degrades far less under contention than BF16, because it moves roughly a third of the bytes per token — on a box already serving other models, that matters more than the single-stream figure.

It can actually see

A community NVFP4 of this same base returned !!!!!!!! for every image — including a solid red square — while passing arithmetic, tool calls and 43 tok/s, at 668 downloads. It had quantized the vision tower: 25 ignored layers, none of them vision.

This build holds out 98 vision modules; all 297 vision tensors stay unquantized, and every release is gated on real images, not text canaries.

PASS  VISION solid red     'red'          PASS  arithmetic 19x23   437
PASS  VISION solid green   'green'        PASS  tool call parses   ✓

Quality

Quantization costs accuracy. Same 603 rows, same endpoint, same decode path.

~8× fewer invented identifiers than the untrained base; about half the BF16 margin given back.

Use NVFP4 when throughput or footprint is the constraint. Use BF16 when you need the best accuracy on reading identifiers exactly.

Use

vllm serve KartiOS/Karti-Small-VL-4B-NVFP4 \
  --max-model-len 32768 --kv-cache-dtype fp8 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Recipe

Held out: lm_head · re:.*\.visual\..* (297 tensors) · re:^mtp\..* · re:.*linear_attn\.conv1d\.* — 147 entries, 248 modules quantized.

Calibration is frozen so version-over-version comparisons measure the training, not the quantizer: lmms-lab/flickr30k test, 512 samples, seq 2048, seed 115, multimodal — images through the model's own processor, never text-only. It overlaps none of this program's evaluation sets.

No MTP head in this build, so no built-in speculative decoding; the BF16 has it. Serve the weights directly, never as a LoRA adapter over the base.

---

From Lumbridge.