B0-9B-GGUF

B0-9B — GGUF

Q8_0 (the quant every benchmark number was measured on) and vision mmproj for schneewolflabs/B0-9B.

llama-server -m B0-9B-Q8_0.gguf -ngl 99 -c 8192 --jinja -fa on -np 1 \
    --spec-type draft-mtp --spec-draft-n-max 4 \
    --mmproj B0-9B-mmproj-f16.gguf

Deployment guide (measured)

mradermacher's static and imatrix quants of this model were benchmarked on the 47-case operator tool bench (Q8_0 anchor: 46/47). Highlights:

Rules that came out of the measurement matrix (i1-Q3_K_M, RTX A6000; ratios transfer):

• Always --jinja and -fa on. Tool calling depends on the chat template.
• Offload every layer you can. Quality is unaffected by partial offload, but each CPU layer costs ~20% generation speed at the top of the curve (34 layers: 102 tok/s → 33: 82 → 32: 67 → 30: 50). Prefill degrades much more gently.
• Keep the KV cache at f16. This is a hybrid-attention architecture — KV costs only ~33MB per 1k tokens — and every cache quantization measured (q8/q8, q8/q4, q4/q4) cost 1–2 tool-bench cases while saving little. Quantize cache only when a context target won't otherwise fit.
• The imatrix.gguf in the i1 repo is for re-quantizing, not inference — the i1 GGUFs are self-contained.

6GB VRAM recipes (i1-Q3_K_M, 139MB/layer):

# 32k context, full quality, full speed — ~5.6GB
llama-server -m B0-9B.i1-Q3_K_M.gguf -ngl 99 -c 32768 --jinja -fa on -np 1

# 64k context — ~5.7GB, costs ~2 tool-bench cases
llama-server -m B0-9B.i1-Q3_K_M.gguf -ngl 99 -c 65536 --jinja -fa on -np 1 \
    --cache-type-k q8_0 --cache-type-v q8_0

# <5.5GB usable (display attached): the resurrected 2-bit
llama-server -m B0-9B.i1-Q2_K.gguf -ngl 99 -c 32768 --jinja -fa on -np 1

Vision (--mmproj) adds ~1GB — skip it on 6GB cards.