Ornstein-3.5-9B-V2-ROCmFPX-STRIX_LEAN-GGUF

Ornstein-3.5-9B-V2 ROCmFPX STRIX_LEAN — GGUF

ROCmFPX Q4_0_ROCMFP4_STRIX_LEAN quant of GestaltLabs/Ornstein-3.5-9B-V2 (Qwen3.5 9B + RLVR/GRPO post-training, 9.2 B params, native multimodal with vision tower, single-file MTP head for speculative decoding).

Built with charlie12345/ROCmFPX on a Radeon RX 9060 XT 16 GB (gfx1200), ROCm 7.2.3, NixOS 25.11. Quantized 2026-06-28 with build commit 11d76c2.

This is not a stock llama.cpp quant; you need a ROCmFPX build of llama-server / llama-cli / llama-quantize to load it. The ROCmFP4 weight format is unknown to stock llama.cpp and will fail with unknown quantization.

Multimodal note: this is a vision-capable model. To use the vision tower, also load the mmproj-ornstein-v2-f16.gguf (~921 MB) companion file via --mmproj <path>. The companion is published separately by GestaltLabs in Ornstein-3.5-9B-V2-GGUF. Vision is verified to work in mesh_eval (1×1 test pixel identified as "Red").

Scope of these benchmarks — read this first

These numbers are a light baseline, not a thorough ROCmFPX evaluation. The mesh's bench framework is built for production agent workload regression-detection on the local stack, not for the kind of multi-axis sweep that upstream quant maintainers typically publish. Specifically:

• Harness scope is bounded. The numbers below come from the mesh's mesh_eval (6 tests, 4 deterministic + throughput + vision) + hermes_loop_eval (5 agent scenarios) + a ctx_scaling_bench run at 4 K → 32 K (64 K+ blocked by harness HTTP timeout, not model capability). That's a regression suite, not a quality benchmark — it answers "does this quant still serve the mesh's agent stack correctly," not "is this the best possible 4-bit ROCmFP4 quant of this model."
• Sample sizes are small. Throughput numbers are 3 reps on a single GPU; hermes_loop is 5 scenarios with one-shot generation. None are powered for statistical significance on a per-token level.
• No perplexity / wikitext / MMLU / GSM8K. The mesh's stack isn't a quality benchmark — those are upstream ROCmFPX's territory. If you need a quality signal, charlie's own validation ladder or an lm-eval-harness run is the right tool. (Note: GestaltLabs's own published GPQA / reasoning numbers on the parent V2 model are 1.00 / 1.00 on the GBS-200 suite — those are upstream's numbers, not ours.)
• Single GPU class. All measurements are on a 16 GB RDNA4 (RX 9060 XT, gfx1200). No Strix unified-memory, no CDNA, no multi-GPU, no Vulkan, no CUDA. Cross-hardware generalization is not implied.
• No human eval. "Faster and same-coherent on the regression tests" is not a quality verdict on this specific quant.

What this IS good for: a quick signal that the quant (a) loads, (b) runs at sane throughput, (c) doesn't break the mesh's agent tool-calling, (d) handles the vision path, (e) scales predictably with context. What this is NOT good for: claiming "this is the best quant of this model," reproducing academic benchmark results, or substituting for upstream's validation work.

For a rigorous view, the parent repo GestaltLabs/Ornstein-3.5-9B-V2 (which itself includes a published GBS-200 benchmark table) and the model's stock GGUF variants on GestaltLabs/Ornstein-3.5-9B-V2-GGUF and mradermacher/Ornstein-3.5-9B-V1.5-i1-GGUF are the place to look.

What we measured

Hardware: Node B, AMD Ryzen 9 5900XT 16-core, Radeon RX 9060 XT 16 GB (gfx1200), ROCm 7.2.3, NixOS 25.11 Software: charlie12345/ROCmFPX main @ 11d76c2 Source GGUF: ornstein-v2-f16.gguf (F16, 18.4 GB — includes the single-file MTP draft head baked into the same file) sourced from GestaltLabs/Ornstein-3.5-9B-V2-GGUF Companion file for vision: mmproj-ornstein-v2-f16.gguf (~921 MB) — not part of this upload, see GestaltLabs repo Same-stack comparison: none — the only same-source Q3_0_ROCMFPX quant for Ornstein we have is a baseline reference, not a same-harness A/B (different sampler settings, different time). The headline below is the model itself, not a comparison.

Throughput (mesh_eval, 4 K ctx, MTP-ON, turbo4 KV, rep_pen=1.1)

3 reps of 256-token completion, gen t/s mean 47.2 ± 0.4 (45.3, 47.5, 47.4 individual reps). This matches Vibetuned STRIX_LEAN's 47-48 t/s range on the same Node B card despite Ornstein being 40 % smaller (4.84 GB vs 7.0 GB).

Agent / loop validation (raw JSON: raw-hermes-loop-ornstein-v2-strix-lean-reppen1.1.json)

mesh_eval.py 4 deterministic + vision (raw-mesh-eval-ornstein-v2-strix-lean.json):

hermes_loop_eval.py 5 scenarios with rep_pen=1.1 (raw-hermes-loop-ornstein-v2-strix-lean-reppen1.1.json):

rep_pen=1.1 is required for this model (and for Fable5, the other RLVR/GRPO-trained model in the mesh's stack). Without it, the model loops on chained and multi_step scenarios — the same tool-loop pattern the mesh's Fable5 work hit. With rep_pen=1.1 the model passes 5/5. The baseline (default sampler) was 3/5; that raw JSON is included as raw-hermes-loop-ornstein-v2-strix-lean-baseline.json for reference. The 2 missing scenarios are tool-loop failures, not quant defects.

Context scaling (raw JSON: ctx-scaling-ornstein-v2-strix-lean-20260628-212553.json)

Findings:

• 32 K prompt processing holds at 1140 pp t/s — the model handles 32 K comfortably on a 16 GB card with KV offload.
• Decode throughput holds at 50 t/s at 32 K (matches Ornith 9B and Vibetuned 14B on the same card).
• 64 K+ ctx scaling is harness-limited, not model-limited. The harness's 120 s urlopen timeout blocks measurement before the model can finish. Server health at 128 K is verified separately (with cram=24576, server is healthy at 71 % VRAM and processes 64 K+ prompts in ~7 min). The ctx_scaling_bench harness needs a longer HTTP timeout for proper 64 K+ measurement — that's a separate follow-up, not a model issue.
• The 128 K test that initially OOM'd was due to cache-ram=8192 being too small for 128 K; bumping to cache-ram=24576 (24 GB DDR4 budget on Node B's 48 GB) resolves it.

KV cache type (head_dim=128, same as Ornith + Vibetuned)

The mesh's KV-type sweep was run on the head_dim=128 Qwen family. turbo4 is the production default for any head_dim=128 model in the ROCmFPX build: -0.7-1.1 GB VRAM, same throughput vs q8_0. See the Ornith 9B ROCmFPX STRIX_LEAN repo for the full sweep data. turbo3/4 are TheTom's turboquant types, absorbed into ROCmFPX main via PlunderStruck commits d859c9e + d0141e8.

Quick start

# Build llama.cpp with ROCmFPX
git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
cmake -S . -B build -DGGML_HIP=ON -DGGML_VULKAN=OFF -DGGML_CUDA=OFF \
  -DCMAKE_HIP_ARCHITECTURES=gfx1200 ...
cmake --build build --target llama-server llama-cli llama-quantize

# Download the mmproj companion for vision (separately published by GestaltLabs)
# wget https://huggingface.co/GestaltLabs/Ornstein-3.5-9B-V2-GGUF/resolve/main/mmproj-ornstein-v2-f16.gguf

# Serve (131 072 ctx, turbo4 KV for head_dim=128, fa=on, MTP-ON, rep_pen=1.1)
./build/bin/llama-server \
  -m Ornstein-3.5-9B-V2-ROCmFPX-STRIX_LEAN.gguf \
  --mmproj mmproj-ornstein-v2-f16.gguf \
  -np 1 -c 131072 \
  -ctk turbo4 -ctv turbo4 \
  -kvo -cram 24576 -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 --spec-draft-p-min 0.75 \
  --repeat-penalty 1.1

--spec-type draft-mtp is the correct flag, not mtp (the --spec-type mtp form in the upstream GestaltLabs HF model card is a typo; the ROCmFPX llama-server rejects mtp with a list of valid types). This is the same typo pattern that hit the SABER card upstream.

Reproduce the quant

# Source (F16 GGUF with MTP head baked in, from GestaltLabs)
SRC=/mnt/e/llms-models-data/ornstein/ornstein-v2-f16.gguf

# ROCmFPX llama-quantize (preset is built in; see `llama-quantize --help`)
~/ROCmFPX/build-rdna4/bin/llama-quantize \
  "$SRC" \
  Ornstein-3.5-9B-V2-ROCmFPX-STRIX_LEAN.gguf \
  Q4_0_ROCMFP4_STRIX_LEAN

Quantize time: ~4 min for 18.4 GB F16 source, CPU-only, no GPU required.

Files in this repo

Not in this repo (intentionally): the mmproj-ornstein-v2-f16.gguf (~921 MB) is a separate file published by GestaltLabs in Ornstein-3.5-9B-V2-GGUF. The model card for the parent quant list explicitly says "GGUF" includes the mmproj in the same repo. We don't redistribute it here to avoid a third-party redistribution; download directly from GestaltLabs.

What's NOT in this repo (caveats)

• Stock llama.cpp will not load this file. The ROCmFP4 weight format is unique to charlie12345/ROCmFPX. Use that fork's llama-server/llama-cli/llama-quantize.
• No CUDA / non-AMD GPU bench. All measurements are RDNA4 (gfx1200). Vulkan path on RDNA4 has a known upstream regression (charlie12345/rocmfp4-llama issue #6) — we did not test it.
• 64 K+ ctx scaling is harness-limited, not model-limited. The ctx_scaling_bench.py 120 s HTTP timeout blocks measurement at 64 K. The model itself handles 128 K ctx (verified separately with cram=24576). Proper 64 K+ numbers will require a harness fix (longer timeout or async polling) — that's a separate follow-up.
• The source GGUF is GestaltLabs-distributed (per general.quantized_by in the F16 source metadata). The actual parent is GestaltLabs/Ornstein-3.5-9B-V2 (the safetensors model), itself a finetune of GestaltLabs/Ornstein-3.5-9B-V1.5, itself a finetune of Qwen/Qwen3.5-9B. The chain is: Qwen3.5-9B → V1.5 (SFT) → V2 (DPO + GRPO/RLVR post-training) → GestaltLabs F16 GGUF → our STRIX_LEAN.
• 5 GB minimum VRAM for the GGUF alone; 12 GB with KV offload at 128 K. The mesh's 16 GB card runs it with ~3 GB headroom at 128 K ctx.
• rep_pen=1.1 is mandatory for the agent loop. Without it, the model loops on chained and multi_step (3/5 PASS). This is the same Fable5 tool-loop pattern — a property of the RLVR/GRPO SFT family, not the quant. The fix is universal: add --repeat-penalty 1.1 to the serve command. (Note: this is unusual for an Apache-2.0 release to require; upstream's HF card does not document it. A friendly bug report to GestaltLabs is the right next step.)
• --spec-type mtp is a typo in the upstream HF model card. The correct flag for llama-server is --spec-type draft-mtp. The mtp form is rejected with a list of valid types. This is a separate upstream bug.
• Vision requires the mmproj-ornstein-v2-f16.gguf companion file. Not bundled in this repo; download from GestaltLabs/Ornstein-3.5-9B-V2-GGUF. The model card there labels the mmproj as mmproj-ornstein-v2-f16.gguf and notes it covers image + video input.
• No MTP / speculative-decode sweep on this file beyond the default --spec-draft-n-max 3 --spec-draft-p-min 0.75. The mesh's MTP sweep was on Fable5 (Node D CUDA, +81% decode on Fable5 Q8 + MTP). Ornstein MTP settings may have a different optimal; we used the upstream-recommended values.
• No quality benchmark (perplexity, MMLU, GSM8K). GestaltLabs's own published GPQA / reasoning numbers on the parent V2 model are 1.00 / 1.00 on the GBS-200 suite (per their HF card) — those are upstream's numbers, not ours.

Provenance

• Source model: GestaltLabs/Ornstein-3.5-9B-V2 — 9.2 B params, Qwen3.5 9B base + DPO + GRPO/RLVR post-training, vision tower + MTP head baked in
• Source model license: apache-2.0
• Source GGUF uploader: GestaltLabs (the model authors themselves)
• Companion file: mmproj-ornstein-v2-f16.gguf (~921 MB) in GestaltLabs/Ornstein-3.5-9B-V2-GGUF (NOT in this repo — see "Files in this repo" above)
• Quantizer: charlie12345/ROCmFPX main @ 11d76c2 (2026-06-27)
• Quantizer license: MIT
• Build hardware: Node B, AMD Ryzen 9 5900XT 16-core, Radeon RX 9060 XT 16 GB (gfx1200), ROCm 7.2.3, NixOS 25.11
• Build tooling: NixOS 25.11, ROCm store paths dynamic-discovered. See the meshina repo's references/nixos-rocm-external-build-recipe.md for the build env setup.
• Bench harnesses: scripts/mesh-bench/mesh_eval.py + scripts/mesh-bench/hermes_loop_eval.py + scripts/mesh-bench/ctx_scaling_bench.py from the meshina repo (private)
• Original bench report: raw/benchmarks/2026-06-28-ornstein-charlie-bench/SUMMARY.md in the meshina repo (177 lines, full session record + cross-model comparison + 6 caveats)
• Research note on 27B Ornstein feasibility: raw/research/2026-06-28-ornstein-27b-charlie-size-math.md (concludes 27B Ornstein is not feasible on 16 GB at 128 K; defer to 24 GB+ hardware)

License

• The Ornstein 3.5 9B V2 parent is apache-2.0 (per its HF model card).
• The charlie12345/ROCmFPX quantizer is MIT.
• The GGUF in this repo is a derivative of the apache-2.0 parent, produced with the MIT-licensed quantizer. Both upstream licenses are preserved.