Downloads · 30 days
464
15% of all-time downloads
OrionLLM_GRM-3.2-Sky-GGUF-MTP
thread13/OrionLLM_GRM-3.2-Sky-GGUF-MTP
OrionLLM_GRM-3.2-Sky-GGUF-MTP is a image-text-to-text model from thread13. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product.
<div style=" border: 2px outset 90EE90; background-color: f0fff0; padding: 10px; border-radius: 5px; display: inline-block; font-family: sans-serif; "
All-time downloads
3.2K
Public
Repo size
119 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.gguf119 GB · 100%
At a glance
- Task
- Image-Text-to-Text
- Access
- Public
- Created
- Aug 7, 2026
- Updated
- Aug 17, 2026
- SHA
- 9279f823
Try a prompt
Base models
Model card
From the Hugging Face model README
<div style="
border: 2px outset #90EE90;
background-color: #f0fff0;
padding: 10px;
border-radius: 5px;
display: inline-block;
font-family: sans-serif;
">
llama.cpp gguf files for OrionLLM/GRM-3.2-Sky, including IQ4 quants.
2026-08-17 : added a quick perplexity comparison for recent (August 2026) Qwen3.5 35B A3B-alike models, spoiler:
- it is entirely a personal preference, but for coding-related tasks I would go for
Kwaipilot/KAT-Coder-V2.5-DevorOrionLLM/GRM-3.2-Sky.
Table of Contents
<!-- todo: - [GRM-3.2-Sky-IQ4_XS-MTP](#llama.cpp-invocation-examples-GRM-3.2-Sky-IQ4_XS-MTP) - vision options -->About
- (WIP) more GGUF quantizations will be coming after some tests
- (WIP) both the model card and the repository structure are subject to change in the next coming days
This repo shall contain some "classic" gguf format quantizations for OrionLLM/GRM-3.2-Sky model, which reports (as of 2026-08-08) exceptional coding and agentic benchmarks (cited below), including two "IQ4" quants -- IQ4_XS and IQ4_NL quants, bult with an "importance matrix" based on a sample from deepmind/code_contests dataset -- see below.
The GGUFs were made with llama.cpp versions 10297 - 10309 ( build ids: 10297 - (6a32c29a7) ; 10309 - (82bb48500)`),
and then were combined with external MTP layers borrowed from compatible models (a4lg, unsloth)
using Python code by buzz and tooltd.
Every quantization is tested for validity on somewhat non-trivial programming tasks and will be uploaded based on testing progress.
Note: you would also likely want to check out these quantizations:
GRM-3.2-Sky by OrionLLM
OrionLLM/GRM-3.2-Sky shows very impressive performance on agentic and coding benchmarks, scoring a 87.7% result on LiveCodeBench v6 according to their HF model card (as of 2026-08-08):
Detailed Benchmarks
- From the original model card:
Scores are taken from each provider's own published model card, blog post, or system card where available; "—" indicates a score was not publicly reported by that provider at the time of writing. Different labs may use different agent scaffolds when reporting SWE-bench and Terminal-Bench results, so cross-provider comparisons should be read with that caveat.
Quantizations
<!-- # Quantizations -->On MTP layer selection
- https://huggingface.co/DevQuasar/Qwen.Qwen-AgentWorld-35B-A3B-GGUF/discussions/1#6a3c4b123ac3e2e1a9ff7bff
- https://huggingface.co/ivgranite/Qwen-AgentWorld-35B-A3B-GGUF
There were some minor issues using a straightforward convert_hf_to_gguf.py safetensors-to-gguf conversion, which can be worked around by passing some specific options to llama.cpp (see the links above), or using an excellent mtp grafting script created by tooltd; the latter, however, expects a stripped-down "mtp-only" input gguf data, such as the one created by a4lg.
So a much more straightforward way is to simply use convert_hf_to_gguf.py script with --no-nextn option, which then allows us to use the classic conversion script by buzz and add MTP layers from other models (for example unsloth/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf).
Either way, we are using Q8_0 quantization for MTP layers, as a well-known compromise between resources taken (memory), accuracy and speed.
I would also recommend using q8_0 quantization for KV cache, as it worked pretty well in my tests; however, some users report severe loss of accuracy while using this pathway, so your mileage might vary; do your own tests based on your task complexity and required throughput.
On IQ quants
This repository also contains IQ4_XS and IQ4_NL quants, based on now-classic format devised by Iwan Kawrakow. For a change, they are not based on an importance matrix based on wikitext (wikitext-2-raw) or some other general purpose text, but instead they use a custom code-specific importance matrix based on a sample from deepmind/code_contests dataset.
The idea was to make a selection of code examples of moderate difficulty (levels 4, 10-19), so that the importance matrix generation would be based on (a) code and (b) somewhat "practical" code -- that is, not one's simple textbook examples and not too convoluted coding exercises, but rather on the tasks and respective code complexity that one might reasonably expect to see in practice.
In all fairness, that was not exactly what I had in mind but rather the closest approximation one can get to in a highly constrained time window; ideally, I'd prefer the text sample set to contain less creative language (we need the model to understand long boring practical task descriptions) and many more references to external code (as any practical programming would normally have much less complex algorithm implementations and would rather refer to some venerable library code containing these); however, one very important feature of code_contests dataset is that it contains correct code, and that might already be a good start -- basically I expect to either see that using a domain-specific imatrix is a good, or, on the contrary, is a rather bad idea, and then we can make some further steps based on that.
Finally, a few quick words on mostly concentrating on "average" complexity levels -- the reasoning behind this was simple: (a), I believe we are far beyond the point when a model might have difficulties generating any simple code (what's your name: "Peter"; hello, Peter); (b), if we want to keep track of the code that we generate, then the classic rule is to avoid writing the most complex code that you can, as by definition you will never be able to debug it; therefore -- our goal are middle grounds.
On model vision
This is a vision-capable model, and both BF16 and Q8_0 quantizations are provided. To apply, pass an mmproj gguf file to llama.cpp via --mmproj argument:
llama-server \
... \
--model mmproj.OrionLLM-GRM-3.2-Sky.bf16.gguf \
...
NB(1). If, like me, you have a habit of carefully studying service logs after changing settings for a specific service, you would very likely notice something like the following warning:
[50549] 0.07.765.250 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
[50549] 0.07.765.252 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
[50549] 0.07.765.252 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842
In other words, it is advised to give enough resolution to our model' visual input:
llama-server \
... \
--model mmproj.OrionLLM-GRM-3.2-Sky.bf16.gguf \
--image-min-tokens 1024 \
...
NB(2). As any other layer, multimedia input takes resources; so if your use case is not heavily vision oriented (such as webdev),
you might want to experiment with telling llama.cpp to keep the vision layers off vram and leave it primarily for text-based inference:
llama-server \
... \
--model mmproj.OrionLLM-GRM-3.2-Sky.Q8_0.gguf \
--no-mmproj-offload \
...
llama.cpp invocation examples
This likely won't be too bad as a general suggestion -- but can be improved further by tailoring for a specific vram amount or device type; some examples are below:
llama-server \
--model ${GGUF_FILENAME} \
--flash-attn on \
--n-gpu-layers-draft all \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--temperature 1.0 \
--top-k 20 \
--top-p 0.95 \
--min-p 0 \
--reasoning-preserve \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--cache-type-k-draft q8_0 \
--cache-type-v-draft q8_0
Normally, Qwen-3.5-35B-A3B model family quantizations would work best with 20+ Gb of vram; in below examples, we'll be mostly targeting a 24 Gb vram target, but it is possible to fit 20 - 22 Gb of vram by using -ctk q8_0 -ctv q8_0 -c ${context_size}, for low enough (e.g. 32k - 40k) values of ${context_size}.
When we let kv cache offload to main memory, one might notice that the inference speed drops significantly (e.g. down to 50% or less compared to a perfect vram memory fit). However, it is often the only way to work with long contexts, and so could be a perfectly reasonable step to take -- as always, "your mileage may vary".
NB. For all of the examples below we assume a single user who mostly works on sequential and related tasks; your work pattern might be different, so check what works best for you.
llama.cpp invocation examples - GRM-3.2-Sky-IQ4_XS-MTP
IQ4_XS (aka "4.25 bpw") quantization + Q8_0 MTP layers shall fit into 24 Gb of vram with settings like these:
${LLAMA_SERVER} \
--model OrionLLM.GRM-3.2-Sky.IQ4_XS.mtp.gguf \
--no-mmproj \
--flash-attn on \
--n-gpu-layers all \
--kv-offload \
--kv-unified \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--temperature 1.0 \
--top-k 20 \
--top-p 0.95 \
--min-p 0 \
--reasoning auto \
--reasoning-preserve \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--cache-type-k-draft q8_0 \
--cache-type-v-draft q8_0
NB: check out choosing optimal mtp settings section below in regards to choosing best fit values for --spec-draft-n-max <n> option.
llama.cpp invocation examples - GRM-3.2-Sky-IQ4_XS-MTP (vision)
IQ4_XS (aka "4.25 bpw") quantization + Q8_0 MTP layers + vision layers (in BF16) shall fit into 24 Gb of vram with settings like these:
${LLAMA_SERVER} \
--model OrionLLM.GRM-3.2-Sky.IQ4_XS.mtp.gguf \
--mmproj mmproj.OrionLLM-GRM-3.2-Sky.bf16.gguf \
--ctx-size 200000 \
--flash-attn on \
--n-gpu-layers all \
--kv-offload \
--kv-unified \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--temperature 1.0 \
--top-k 20 \
--top-p 0.95 \
--min-p 0 \
--reasoning auto \
--reasoning-preserve \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--cache-type-k-draft q8_0 \
--cache-type-v-draft q8_0
NB: check out choosing optimal mtp settings section below in regards to choosing best fit values for --spec-draft-n-max <n> option.
Appendix
On choosing the optimal mtp settings for your case
As for --spec-draft-n-max 4 setting above -- the exact number would obviously depend on your GPU model.
You might want to do a log scale search by trying
spec-draft-n-max := 1,2,4,8,...(unlikely more) while the inference speed grows or stays the same,- and then using binary search to narrow down the exact value that works best for you on your typical tasks;
- for a quick control check, your server log will show something like:
- nb: showing results for the same model running two consequitive coding tasks
[37813] 812.35.434.815 I slot launch_slot_: id 0 | task 56272 | processing task, is_child = 0
[37813] 812.37.748.230 I slot print_timing: id 0 | task 56272 | prompt eval time = 310.31 ms / 177 tokens ( 1.75 ms per token, 570.40 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id 0 | task 56272 | eval time = 2002.97 ms / 400 tokens ( 5.01 ms per token, 199.70 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id 0 | task 56272 | total time = 2313.28 ms / 577 tokens
[37813] 812.37.748.235 I slot print_timing: id 0 | task 56272 | graphs reused = 52595
[37813] 812.37.748.238 I slot print_timing: id 0 | task 56272 | draft acceptance = 0.95482 ( 317 accepted / 332 generated), mean len = 4.82
[37813] 812.37.749.091 I slot release: id 0 | task 56272 | stop processing: n_tokens = 29120, truncated = 0
[37813] 812.38.114.222 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 0.988 (> 0.100 thold), f_keep = 1.000
[37813] 812.38.115.131 I slot launch_slot_: id 0 | task 56358 | processing task, is_child = 0
[37813] 812.38.767.224 I slot print_timing: id 0 | task 56358 | prompt eval time = 440.40 ms / 341 tokens ( 1.29 ms per token, 774.29 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id 0 | task 56358 | eval time = 211.57 ms / 20 tokens ( 10.58 ms per token, 94.53 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id 0 | task 56358 | total time = 651.97 ms / 361 tokens
[37813] 812.38.767.229 I slot print_timing: id 0 | task 56358 | graphs reused = 52603
[37813] 812.38.767.232 I slot print_timing: id 0 | task 56358 | draft acceptance = 0.33333 ( 12 accepted / 36 generated), mean len = 2.33
As you can see, we could have also tried --spec-draft-n-max 5 and see if that would improve the benchmarks: basically as long as "mean len" frequently shows values
which are greater than your spec-draft-n-max parameter -- you can still increment it by at least 1 and see if it improves your statistics (and waiting times).
Also, in some scenarios there could be some use for additional settings like --spec-draft-p-min 0.5.
I am no expert, but in my tests I had some luck with first determining the optimal --spec-draft-n-max value,
and then seeing if raising up the acceptance min level (and/or the "split" values such as --spec-draft-p-split) will improve things a little more.
- Task
- Image-Text-to-Text
- Created
- Aug 7, 2026
- Updated
- Aug 17, 2026