Ornstein-3.5-9B-V2-GGUF

Ornstein 3.5 9B — V2 · GGUF

GGUF quantizations of GestaltLabs/Ornstein-3.5-9B-V2 — the reinforcement-learning post-training (V2) of Ornstein 3.5 9B. Each quant has a sibling .sha256 checksum, and a separate vision projector (mmproj) ships the multimodal tower for image/video input.

Files

Usage (llama.cpp)

Text:

llama-cli -m ornstein-v2-Q4_K_M.gguf -p "Derive the variance of a sum of two correlated random variables."

Multimodal (image/video) — add the vision projector:

llama-mtmd-cli -m ornstein-v2-Q4_K_M.gguf \
  --mmproj mmproj-ornstein-v2-f16.gguf \
  --image picture.jpg -p "Describe this image."

Quality and speed scale with quant size; Q4_K_M is a strong default for ~8 GB of VRAM/RAM.

Speculative decoding (MTP)

Every quant embeds the model's native multi-token-prediction (MTP) draft head (GGUF block_count 33), so self-speculative decoding runs from a single file — no separate draft model needed:

llama-cli -m ornstein-v2-Q4_K_M.gguf --spec-type draft-mtp \
  -p "Write a Python function is_prime(n)."

Support This Work

I'm a PhD student in visual neuroscience at the University of Toronto who also happens to spend way too much time fine-tuning, merging, and quantizing open-weight models on rented H100s and a local DGX Spark. All training compute is self-funded — balancing GPU costs against a student budget. If my uploads have been useful to you, consider buying a PhD student a coffee. It goes a long way toward keeping these experiments running.

Support on Ko-fi

License

Apache 2.0 — inherited from the Qwen 3.5 9B base release.