Spark-X2.5-4B-build-cpp

Spark-X2.5-4B — Ready-to-Run llama.cpp Build & GGUF Package (Colab T4 / CUDA)

Everything needed to run iFlytek Spark-X2.5-4B on an NVIDIA T4 (16GB) with llama.cpp: a prebuilt CUDA binary (XHToken fork — the spark2_5 architecture is not supported by mainline llama.cpp), Q8_0 / Q4_K_M model weights, a 1.7B draft model (boosts the Q8_0 config by +41%, 0.89 acceptance), and battle-tested one-click scripts (256k-context OpenAI-compatible API + public tunnel).

Verified end-to-end on 2026-09-06: streaming, native tool calling (--jinja), and a real agent workload (84-tool zagent, 12 consecutive tool calls, 240k prompt tokens, zero failures).

What's in this repo

Quick start (~3 minutes on a fresh T4 Colab)

# Cell 1 — mount Drive
from google.colab import drive; drive.mount('/content/drive')

# Cell 2 — restore prebuilt binary (public repo, no token needed, ~1 min)
pip install -U huggingface_hub >/dev/null
hf download gasschina/Spark-X2.5-4B-build-cpp llama.cpp-spark-build-t4-cuda.tar.gz --local-dir /content/
mkdir -p /content/drive/MyDrive/spark-t4
tar xzf /content/llama.cpp-spark-build-t4-cuda.tar.gz -C /content/drive/MyDrive/spark-t4/

# Cell 3 — fetch scripts & launch (downloads weights to local disk + starts API + tunnel + self-test)
hf download gasschina/Spark-X2.5-4B-build-cpp --include "scripts/*.sh" --local-dir /content/s4b-scripts
cp /content/s4b-scripts/scripts/*.sh /content/drive/MyDrive/spark-t4/
bash /content/drive/MyDrive/spark-t4/oneclick_start.sh

The launcher prints the local endpoint, public tunnel URL, and API key when done. Model weights live on Colab local disk (/content/spark-t4-models), never eating into the 15GB Drive quota; the build and scripts persist on Drive across sessions.

Default config (v6): Q8_0 + 1.7B draft @ 256k (reverted 2026-09-06)

Draft policy is automatic by quant tier: QUANT=Q8_0 attaches the 1.7B draft (acceptance 0.89, 52.3 tok/s, 11.8GB); 4-bit runs bare. Override with SPEC_TYPE=draft-simple / SPEC_TYPE=none.

QUANT=Q8_0 bash oneclick_start.sh      # quality + draft: 52.3 tok/s, 11.8GB
CTX=500000 bash oneclick_start.sh      # 500k long-context, same speed, 10.1GB
QUANT=Q4_K_M CTX=786432 bash oneclick_start.sh   # 768k extreme, 14.3GB

1M context physically does not fit on a T4: 1M KV@q4_0 alone needs ~13GB before weights. Single slot (-np 1): concurrent requests queue behind long agent replies. On agent platforms (aicq.me etc.) prefer streaming and keep max_tokens ≤ 2000.

Benchmarks (T4, greedy, 384-token generations, /completion timings)

Takeaway: the T4 is memory-bandwidth-bound. Drafting is a distribution-matching game — with a near-lossless Q8_0 target the 1.7B sibling hits 0.89 acceptance (+41%), but a 4-bit target's argmax flips too often and drafting goes net-negative. Gateway note: the free aitun tunnel terminates non-streaming requests at exactly 120s (reproduced); streaming is immune.

MTP status (why classic drafting)

llama.cpp already ships draft-mtp / draft-eagle3 / draft-dspark speculative types, but they require matching trained heads. Spark-X2.5-4B publishes no MTP/nextn/EAGLE weights (num_nextn_predict_layers is absent from its config), and the fork's spark2_5 backend has no nextn implementation. Classic draft-model speculation with the official same-vocabulary (131,072) 1.7B sibling is the best available option: 0.891 acceptance, mean accepted length 8.13 tokens on the Q8_0 target.

API usage (OpenAI-compatible)

import requests
BASE = "http://127.0.0.1:8100"   # or your public tunnel URL
KEY  = "sk-..."

r = requests.post(f"{BASE}/v1/chat/completions",
    headers={"Authorization": f"Bearer {KEY}"},
    json={"model": "spark-x2.5-4b",
          "messages": [{"role": "user", "content": "Hello"}],
          "max_tokens": 512})
print(r.json()["choices"][0]["message"]["content"])

• Streaming: add "stream": true (SSE) — strongly recommended behind tunnels/gateways
• Thinking mode (server-side off by default): add "chat_template_kwargs": {"enable_thinking": true}
• Tool calling: pass a standard OpenAI tools array (--jinja enables native function calling)

Why this build exists

Mainline llama.cpp does not support the spark2_5 architecture (ggml-org/llama.cpp issue #28300). This package ships a CUDA build of the XHToken fork (SM 7.5 / CUDA 12.x) so you can skip the 40–60 minute compile. Weights: Q8_0 / Q4_K_M from the community i1 quant; the 1.7B draft quantized in-house (Q8_0) from the official F16 GGUF.

中文速览

讯飞星火 Spark-X2.5-4B 在 Colab T4 上的完整 llama.cpp 运行包。默认配置 v5 = Q4_K_M 裸跑 @256k 上下文 + 思考默认关（实测 48.75 tok/s，显存仅 6.6GB）——为 aicq.me 等智能体平台调优：aitun 隧道对非流式请求有 120s 硬超时（开思考必 504），故思考默认关闭、单请求可传 enable_thinking:true 恢复；草稿投机解码只对 Q8_0 有正收益（+41%，接受率 0.89），QUANT=Q8_0 时自动挂载，4bit 档实测反而慢一半已自动关闭。API 为 OpenAI 兼容格式，模型名 spark-x2.5-4b，智能体平台建议开流式、max_tokens ≤2000。