Raptor-0.6-4B-GGUF

<p align="center"> <a href="https://osaurus.ai"><img src="./osaurus-x-banner.png" alt="Osaurus AI"></a> </p>

Raptor 0.6 — GGUF

imatrix-calibrated GGUF builds of Raptor 0.6, a lightly tuned Spark-X2.5-4B — the format Osaurus for Windows runs.

The release write-up — what happened to Raptor 0.5, why the base changed, and how 0.6 was built and measured — is at osaurus.ai/models/raptor-06.

Both quants were produced with the importance matrix applied. Measured against the model's own bf16 weights on a held-out corpus (100 chunks x 512 tokens) that is disjoint from the imatrix corpus — 9,166 overlapping rows were dropped and the eval asserts the result is non-empty rather than trusting the paths.

Reference perplexity of the bf16 source on that held-out text: 2.1935 ± 0.0129.

Osaurus for Windows — now in beta

Osaurus, the harness Raptor is built for, now runs on Windows 11, and on Windows it serves open models as GGUF — which is what this repo is for. Osaurus for Windows is a native app (x64 and ARM64, no Electron), free, with no account and no telemetry: open models run entirely on your PC, and nothing leaves it unless you point an agent at a cloud provider. Agents read your files, run tools, and keep working while you do something else — the same harness as the Mac app.

Get it: join the beta at osaurus.ai/windows. Leave an email, the private install link arrives by email in batches as places open, and beta testers get a Slack channel with a direct line to the engineers. Beta means beta: the core is there — local models, cloud providers, agents that read files and run tools — some macOS features are not on Windows yet, and first-run model downloads and less common GPU drivers can be rough.

What you need to run models on-device, from the requirements on that page:

At 3.4 GB (Q6_K) or 2.6 GB (Q4_K_M) these files fit comfortably on any 12 GB card. The Windows beta's runtime and harness carry the spark2_5 architecture, so these files load there as-is — pick Raptor 0.6 from the model catalog, or point the app at this repo. The reasoning and tool-call contract below is handled by the harness.

Requires a llama.cpp with spark2_5 support

spark2_5 is a new architecture. Osaurus for Windows carries it. Outside Osaurus it is not in upstream llama.cpp — there is no LLM_ARCH_SPARK2_5 in any upstream release — so stock llama.cpp, Ollama and LM Studio will refuse these files until they carry the architecture. The graph needs the fused q_k_v_proj, the per-head sigmoid attention output gate (g_proj), a gelu MLP, plain RMSNorm with no +1 shift, 27 sliding / 9 full attention at window 512, and a different rope per layer type (full: theta 5e6 over 64 of 256 dims; sliding: theta 1e4 over all 256).

On a Mac, Osaurus serves MLX, not GGUF — use OsaurusAI/Raptor-0.6-4B-JANG_6M there. These GGUF files are for Osaurus for Windows and for llama.cpp-family runtimes.

Serving contract — verified on the shipped Q4_K_M, not assumed

Run llama-server with --jinja so the embedded chat template drives tool and reasoning parsing.

• Sampling defaults ship inside the GGUF: general.sampling.temp = 1.0, general.sampling.top_p = 0.95, general.sampling.top_k = -1. These match the source generation_config.json exactly. No repetition penalty.
• Token ids: bos 0, eos 1, unk 5, pad 2; add_bos_token and add_eos_token both false, because the template emits the sentence markers itself.
• Reasoning is ON by default — the template force-opens the rail, so the generation prompt ends with <|Bot|><think>. <think> and </think> are control tokens (ids 3 and 4). With --jinja, llama.cpp separates the trace into reasoning_content, leaving content clean:

reasoning_content: 'We are asked: "What is 84 * 3 / 2?" ...'
content:           'The expression \(84 \times 3 \div 2\) is evaluated left to right...'

Pass "chat_template_kwargs": {"enable_thinking": false} to close the rail; reasoning_content then comes back empty. A reasoning parser that waits for a literal <think> in the output stream will never see one — the opener is in the prompt.

• Tool calls parse to standard OpenAI JSON. The model emits the Spark XML dialect (<tool_call>NAME<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>) and llama.cpp converts it:

{"finish_reason":"tool_calls",
 "message":{"tool_calls":[{"type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Seoul\"}"}}]}}

• Tools render into the first system block. Adding or removing a tool rewrites the system prefix, so a tool-set change is a full re-prefill, not a suffix append — prefix-cache accounting must treat it that way.
• Reasoning can need more than 8192 output tokens on hard prompts. Budget real input plus output inside the context window.

What the calibration did, and what it did not

The imatrix was captured over 600 chunks x 512 tokens of code, agentic/tool-call transcripts, academic multiple-choice, general chat, Chinese, long-context, science and security text.

Three arms were built at each width and measured against the same bf16 reference, so the effect of each treatment is a controlled comparison rather than an assumption:

• imatrix earns its place at Q4_K_M (0.0399 vs 0.0435, an 8% reduction) and is within noise at Q6_K, where the format is already near-lossless.
• An AWQ fold was built, measured and dropped. Activation-aware scaling was captured on GPU (1,500,201 tokens, 72/72 fold sites, verified function-preserving to 3.6e-07 and to top-1 35/35 on real logits) and folded into the bf16 before conversion. It made the k-quants worse at both widths — it is redundant on top of a k-quant that already does per-super-block scaling with importance weighting, and its channel rescale widens the dynamic range inside a block. Recorded here because a negative result measured with a control is worth more than an untested claim on the card.

Known limits

Raptor 0.6 is a practical checkpoint with documented rough edges — long-context recall is not guaranteed, ordinary agentic mistakes remain, and identity responses are not enforced. The full list is on the JANG_6M card and applies to these weights unchanged.

Provenance

Quantized by Jinho Jang ([email protected]). Base model XHToken/Spark-X2.5-4B, apache-2.0. Built from the same merged bf16 checkpoint as the JANG_6M bundle. First published on 2026-09-11 as Raptor-0.6.1-preview-GGUF and renamed to Raptor-0.6-4B-GGUF for the release; the files are byte-identical, only their names changed, and the old repo name redirects here.

License

Raptor 0.6 is a derivative of XHToken/Spark-X2.5-4B by the SparkLLM Team, which is released under the Apache License, Version 2.0. The full license text is in the LICENSE file in this repository, carried over unchanged from the base model as the license requires.

Raptor 0.6 is distributed under the same Apache License, Version 2.0. In accordance with Section 4 of that license, the changes made to the original work are: (1) a supervised fine-tune of the attention projections (q_k_v_proj and out_proj, rank-4 LoRA merged into the weights; all other tensors unchanged), and (2) quantization of the merged weights to GGUF Q6_K and Q4_K_M with an importance matrix. The tokenizer, chat template and generation defaults are unchanged from the base model. Osaurus, Raptor and the Osaurus banner are trademarks of Osaurus and are not covered by the Apache license.

If you use this model, please also credit the base model:

@misc{sparkx2.5,
    title  = {Spark-X2.5 4B&1.7B: Pushing the Limits of Agentic Capabilities in On-Device Models},
    author = {SparkLLM Team},
    year   = {2026}
}