Downloads · 30 days
9
16% of all-time downloads
Wondernutts/Orion-26B-A4B-v1-int4-ov
Orion-26B-A4B-v1-int4-ov is a text generation model from Wondernutts. Use it when you need the model to write or continue text. It is set up for openvino.
Original fine-tune: TheDrummer. Original model: TheDrummer/Orion-26B-A4B-v1.
Downloads · 30 days
9
16% of all-time downloads
All-time downloads
55
Public
Repo size
16.1 GB
Likes
2
Public
Click a slice to open those files.
.bin16.1 GB · 100%
From the Hugging Face model README
Original fine-tune: TheDrummer. Original model: TheDrummer/Orion-26B-A4B-v1.
OpenVINO conversion by Wondernutts. TheDrummer created the fine-tune; this repository is its compressed OpenVINO derivative, not a new fine-tune. The released configuration describes the Gemma 4 26B-A4B MoE architecture. The upstream card's base-model metadata lists 31B; that metadata is not evidence that this checkpoint has 31B dense architecture.
TheDrummer's focus on creative writing, expressive dialogue, and dynamic roleplay is exactly what drew me to Orion, and it didn't disappoint. Its lively voice and willingness to vary its responses make it a fun alternative to more predictable RP Gemma models, which are amazing but can get locked into familiar sentence structures. In my use, Orion seems less prone to that. I particularly prefer its dynamic feel over the popular Chimera-X, but that's my subjective preference—not a claim that Orion wins every benchmark.
My custom OpenVINO fork brings fast prompt processing to Orion on Linux/B70. I'm still investigating occasional consistency and runaway-generation issues in this conversion; their cause hasn't been isolated.
These values come from the original release's generation_config.json, not from our Discord bot tuning:
| Setting | Shipped value |
|---|---|
Sampling enabled (do_sample) | true |
| Temperature | 1.0 |
| Top-K | 64 |
| Top-P | 0.95 |
| Repetition penalty | Not specified in that file |
| Frequency penalty | Not specified in that file |
| Presence penalty | Not specified in that file |
| Min-P | Not specified in that file |
These are shipped defaults, not a claim that this is the best sampler for every quantization or RP prompt. A server or frontend can override them; omitting a field does not necessarily disable that server's configured penalty.
The original model card specifies the Gemma 4 31B chat template and supports both thinking and non-thinking operation, including custom thinking blocks. Our performance measurements below used thinking off. Preserve the provided tokenizer and chat template rather than substituting a generic ChatML prompt.
For an OpenAI-compatible endpoint, the corresponding explicitly supplied sampling fields are:
{
"temperature": 1.0,
"top_k": 64,
"top_p": 0.95
}
This is a partial request, not a standalone inference command. Confirm that your serving backend accepts and forwards Top-K. Check server defaults separately for the unspecified penalties and Min-P.
Measured September 8, 2026, on the same Linux B70 test card. Model weights were this Orion conversion; no CPU offload was intentionally configured. These are observations from bounded tests, not vendor benchmarks or a hardware ceiling.
| Input tokens | Time to first token | Input tokens / TTFT | Decode throughput | Generated tokens |
|---|---|---|---|---|
| 6,044 | 0.958 s | 6,310 tokens/s | 95.49 tokens/s | 18 |
| 19,544 | 3.636 s | 5,375 tokens/s | 86.94 tokens/s | 18 |
Each row is one run, not an average or confidence interval. Unique prompt prefixes were used to avoid reusing a warmed conversation prefix; prefix caching remained enabled. Reported input counts matched the full prepared prompts, with no context truncation. Input/TTFT includes overhead beyond pure prefill. Decode estimates from only 18 generated tokens are short-sample measurements.
Those recall runs were deterministic (temperature=0) with repetition penalty 1.1. They were not run with TheDrummer's stochastic defaults. Both returned the requested courier, key location and updated password correctly; this does not establish reliable long-context roleplay generally.
OpenVINO build: 2026.4.0-105-31c98eec8ab-prefill-upstream-integration
OpenVINO GenAI: 2026.4.0.0-3308-79bc2469701
GPU plugin SHA-256:
2818d86aa36e34cf7ce3bd5dd2ab27d3660c21baef39ebeb7a360a9c5dea9388
OV_DQGS=128
OV_KV_CACHE_PRECISION=u4
OV_PREFIX_CACHE=8
OV_MAX_BATCHED_TOKENS=16384
OV_MAX_NUM_SEQS=1
OV_CONTEXT_TOKENS=24576
OV_MAX_CTX_TOKENS=24576
OV_THINK=0
OV_NATIVE_VISION=0
MOE_USE_GROUPED_GEMM_PREFILL=1
MOE_MICRO_GEMM_N_HINT=128
MOE_GROUPED_BINARY_LOOKUP=1
These environment names belong to the custom serving stack; they are not universally recognized OpenVINO API properties. OV_PREFIX_CACHE=8 configures the server's 8 GiB cache setting. INT4 weights, U4 KV cache, and DQ128 dynamic quantization grouping are separate settings.
The 24,576-token context configuration is not a claim that arbitrary 24K prompts plus outputs or concurrent users will fit safely. The longest initial measured input was 19,544 tokens. Do not assume the same speeds, memory capacity or compatibility on B50, consumer Arc, Windows, CPU, or a stock runtime. Custom kernels are separate software and are not baked into these model weights.
Initial loading and recall checks passed; harder RP tests showed scene-tracking mistakes and occasional malformed output. The original full-precision Orion weights have not been compared on the same failing prompts, so these results do not isolate training, conversion or runtime as the cause. Native vision has not been qualified; measurements here are text-only.
TheDrummer/Orion-26B-A4B-v1, revision 22e5bcc5ed81c1d5708846f6356ed1270f22f04b.546089b1628c525357245e82079159a3a75973b7 of Wondernuttz/OpenVino-For-Gemma-4.See orion_conversion_manifest.json for conversion-time metadata. Its inference-validation flag was captured before the subsequent tests summarized here; structural validation alone is not proof of inference quality.
This is OpenVINO IR, not GGUF or a Transformers-loadable weight checkpoint. Use the custom serving stack linked above for the measured setup. The model card does not promise a pip-only reproduction of the exact custom binary.
Fine-tune by TheDrummer. OpenVINO conversion and custom runtime work by Wondernutts. See the original model and the included ORIGINAL_MODEL_CARD.md for the author's information and applicable terms. Conversion does not replace the original model's license or usage terms.