Spark-X2.5-1.7B-GGUF

Llamacpp imatrix Quantizations of Spark-X2.5-1.7B by XHToken

Using <a href="https://github.com/XHToken/llama.cpp">XHToken/llama.cpp</a> fork commit <a href="https://github.com/XHToken/llama.cpp/commit/4a3635c32fc9f044c2bde9ebeabf50c7e1ec5991">b10514-4a3635c32</a> (build tag b10514-4a3635c32) for quantization.

Original model: https://huggingface.co/XHToken/Spark-X2.5-1.7B

Companion repo: stornic56/Spark-X2.5-4B-GGUF — same pipeline, same calibration corpus, same fork commit. The family quality floor study below compares both.

Model details:

• Parameter count: 1.7B
• Input support: text
• imatrix: yes - details

How to run

Update: since llama.cpp b10828 (PR #27868),spark2_5 is supported in mainline llama.cpp — any recent build works for inference with these files. Compatibility was verified on this exact quant set with mainline b10828-3ad1ba733: Q4_K_M and IQ3_M(the most implementation-sensitive quant) load cleanly and pass the same coding coherence test used inthe quality gate — see reproducibility/mainline-b10828-compat.log.

Prompt format

The chat template is embedded in every GGUF; run with --jinja and it is applied automatically. The rendered format is:

<｜start▁of▁sentence｜><|System|>
{system_prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|User|>
{prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|Bot|>
<think>

The model is a thinking model by default: reasoning is emitted inside <think>...</think> before the final answer. The embedded template also supports tool calling, matching the upstream chat_template.jinja.

Don't know which to choose? Grab Spark-X2.5-1.7B-Q4_K_M.gguf (1.03GB) - usually a good mix of size and performance. Download instructions available here.

Available files:

Q3_K_M is intentionally absent: it failed the quality gate on this model (see Family quality floor study below). IQ3_M, which passed, is the smallest recommended quant.

Downloading using the Hugging Face CLI

<details> <summary>Click to view download instructions</summary>

First, make sure you have the Hugging Face CLI installed:

pip install -U "huggingface_hub[cli]"

Download a specific file:

hf download stornic56/Spark-X2.5-1.7B-GGUF --include "Spark-X2.5-1.7B-Q4_K_M.gguf" --local-dir ./

</details>

How to run

These quants require the XHToken llama.cpp fork. Mainline llama.cpp does not implement spark2_5 and will refuse to load these files.

git clone https://github.com/XHToken/llama.cpp.git && cd llama.cpp
# Intel/AMD GPU via Vulkan: add -DGGML_VULKAN=ON
# NVIDIA GPU:               add -DGGML_CUDA=ON
# CPU only:                 plain build works
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build build --config Release -j 8

Interactive chat (template is embedded, --jinja applies it):

./build/bin/llama-cli -m Spark-X2.5-1.7B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja -cnv

OpenAI-compatible server:

./build/bin/llama-server -m Spark-X2.5-1.7B-Q4_K_M.gguf -ngl 99 -c 16384 --jinja

Recommended sampling from the upstream model card: temp 1.0, top_p 0.95, top_k -1. Use --temp 0 for deterministic output.

Other runtimes: Ollama and LM Studio can run these files, but only with builds backed by the same fork - see the "Ollama" and "LM Studio" sections of the official model card for the build steps.

Flash Attention

The fork enables FlashAttention automatically for the sliding-window layers of spark2_5. No extra flag is needed.

imatrix

All quants below bf16 were made using the imatrix option, computed with llama-imatrix from this fork. The calibration corpus is the wikitext-2-raw-v1 train split (Salesforce/wikitext), written to disk verbatim and in order by reproducibility/calibration.py. The corpus file is included in this repo: calibration_data.txt.

The imatrix is available here: Spark-X2.5-1.7B-imatrix.gguf.

<details> <summary>Calibration details</summary>

{
  "dataset": "Salesforce/wikitext",
  "config": "wikitext-2-raw-v1",
  "split": "train",
  "rendering": "raw text, no chat template, no special tokens",
  "chunk_size": 512,
  "chunks": 128,
  "tokens_consumed": 65536,
  "consumption_order": "sequential from file start (deterministic)",
  "threads": 8,
  "final_perplexity": "141.2760 +/- 3.43585",
  "imatrix_format": "GGUF (fork default)",
  "notes": "corpus is plain prose; a future revision may render tool-calling and reasoning conversations through the chat template, as done by other maintainers"
}

Single clean imatrix run (determinism across runs was demonstrated in the 4B project with this exact pipeline and corpus; see the companion 4B repo). Run log: reproducibility/imatrix-run.log.

</details>

Which file should I choose?

<details> <summary>Click here for details</summary>

The first thing to figure out is how big a model you can run. If you want the model running as fast as possible, fit the whole thing in VRAM: aim for a file 1-2GB smaller than your GPU's total VRAM, leaving room for the KV cache (this model's hybrid attention keeps the cache small, which helps). If you want maximum quality and can tolerate partial CPU offload, add your system RAM and VRAM together and pick a file 1-2GB smaller than that total.

Next, K-quant or I-quant? On NVIDIA (CUDA) and Apple silicon, I-quants (IQX_X) generally offer better quality per bit below Q4. On Intel Arc via Vulkan (Mesa), our measurements show the opposite for generation speed: K-quants are faster at equal bit-width, while I-quants win on file size - see the benchmarks below. Either way, both families were validated for coherence in this release.

</details>

Quality Gate (deterministic test, temp 0, seed 1)

All published quants produce a complete, logically correct solution to a Python is_prime coding prompt. Excluded quants and their failure modes:

• Q3_K_M — paragraph-level repetition loop in the reasoning trace, never reached a final answer even with 2× token budget.
• Q2_K — repetition loop + broken logic (labels 27 and 33 as prime).
• IQ2_M — hard repetition loop + hallucinated terminology.

Raw outputs: reproducibility/iq2_m_q2_k_family_study.log

Family quality floor study: 4B vs 1.7B

Same calibration data, same protocol, same fork:

Conclusion: the usable quality floor scales up as models shrink. At 4B the real floor sits between 2 and 3 bits (IQ2_M viable); at 1.7B it is exactly 3-bit and specifically the imatrix-quantized IQ3_M — the K-quant Q3_K_M at similar bpw fails while IQ3_M passes, so --imatrix quantization is not optional for small models in this family.

Benchmarks

Measured with llama-bench from the same fork commit, Intel Arc B580 (Battlemage, Mesa Vulkan driver), -ngl 99 -t 4, 3 runs. Raw log: reproducibility/benchmarks_gpu.txt.

Vulkan (Intel Arc B580 12GB)

• On Vulkan, K-quants generate faster than IQ quants (as with the 4B).
• IQ3_M is an interesting exception: it is faster than Q6_K, Q8_0 and IQ4_NL in generation on this GPU despite the lowest bpw.

CPU (i3-12100F 4C/8T, -t 8, r=3)

Backend-dependent trade-off: IQ3_M is fast on Vulkan but slow on CPU (generic dequant kernels). If you run CPU-only, prefer Q4_K_M / IQ4_NL.

Validation

• test-llama-archs -a spark2_5 from the fork: OK on CPU (NMSE 0.00e+00), Vulkan (NMSE 9.23e-08) and meta buffers. Roundtrip: SKIP is expected for this architecture. Raw log: reproducibility/test-llama-archs.log
• Every published quant passed a deterministic coding coherence test (--temp 0 --seed 1, 700-token budget): complete, logically correct final answer, no repetition. The excluded quants (Q3_K_M, Q2_K, IQ2_M) and the 1500-token Q3_K_M re-run are documented with raw outputs in reproducibility/iq2_m_q2_k_family_study.log.
• bf16 sanity check passed before quantization (coherent reasoning + answer on CPU)

Reproducibility

Everything needed to rebuild these files bit-for-bit is in the repo:

Exact commands:

python convert_hf_to_gguf.py Spark-X2.5-1.7B --outfile Spark-X2.5-1.7B-bf16.gguf --outtype bf16

./build/bin/llama-imatrix \
  -m Spark-X2.5-1.7B-bf16.gguf \
  -f calibration_data.txt \
  -o Spark-X2.5-1.7B-imatrix.gguf \
  -c 512 --chunks 128 -t 8

./build/bin/llama-quantize --imatrix Spark-X2.5-1.7B-imatrix.gguf \
  Spark-X2.5-1.7B-bf16.gguf Spark-X2.5-1.7B-Q4_K_M.gguf Q4_K_M

ARM/AVX information

llama.cpp automatically repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines, covering Q4_0, IQ4_NL and most K-quants. No special quant choice is needed for CPU inference.

Credits

Thanks to the XHToken/SparkLLM team for releasing the model and maintaining the llama.cpp fork, and to bartowski for the quantization card format this repo follows.

License & attribution

The upstream model is released under Apache License 2.0; these derivative quantizations inherit it. Preserve upstream attribution when redistributing. This is an unofficial community release, not endorsed by XHToken/SparkLLM.