Downloads · 30 days
21
10% of all-time downloads
FardoX/sasori-qwen3-14b-tq3p
sasori-qwen3-14b-tq3p is a machine learning model from FardoX. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Ternary quantization of Qwen/Qwen3-14B by sasori: every linear weight is the sum of K=3 ternary planes ({-1,0,+1}) with f16 scales per group of 256 → 6.1875 bits/weight. Embeddings, the output head, norms and biases s…
Downloads · 30 days
21
10% of all-time downloads
All-time downloads
210
Public
Repo size
27.3 GB
Likes
1
Public
Click a slice to open those files.
.gguf26.7 GB · 98%
From the Hugging Face model README
Ternary quantization of Qwen/Qwen3-14B by sasori: every linear weight is the sum of K=3
ternary planes ({-1,0,+1}) with f16 scales per group of 256 → 6.1875 bits/weight. Embeddings,
the output head, norms and biases stay F16 by design. Runs on CPU only, no GPU, no Python.
| file | what it is | size |
|---|---|---|
Qwen3-14B-TQ3P-g256-EF512.gguf | the model — start here. K=3 g=256 + error-feedback calibration | 13,338,008,992 B (12.42 GiB) |
Qwen3-14B-TQ3P-g256.gguf | same format, data-free (no calibration) | 13,338,008,992 B (12.42 GiB) |
sasori-runtime-x86_64-avx2.tar.gz | patched llama.cpp binaries (required, see below) | 15 MB |
Both models are byte-for-byte the same size, format and speed — error-feedback changes only which ternary codes were chosen, never the bit budget. Use the EF one.
Every command below was executed on a bare ubuntu:24.04 container with no gcc, no pip, no curl,
no libgomp, RAM capped at 17 GiB — nothing else was installed.
# 1. download (~90 s for the model on a fast link)
wget https://huggingface.co/FardoX/sasori-qwen3-14b-tq3p/resolve/main/sasori-runtime-x86_64-avx2.tar.gz
wget https://huggingface.co/FardoX/sasori-qwen3-14b-tq3p/resolve/main/Qwen3-14B-TQ3P-g256-EF512.gguf
# 2. extract the runtime
tar xzf sasori-runtime-x86_64-avx2.tar.gz
# 3a. one-shot answer. NOTE the /no_think prefix: without it the model spends the whole
# token budget "thinking" and gets cut off before answering (see the section below).
./sasori-runtime/bin/llama-cli -m Qwen3-14B-TQ3P-g256-EF512.gguf -c 8192 -t $(nproc) -n 256 -st \
-p "/no_think A shop sells pencils at 3 reais each. How much do 7 pencils cost?"
# 3b. interactive chat (this is the DEFAULT — without -st you land in a `>` prompt)
./sasori-runtime/bin/llama-cli -m Qwen3-14B-TQ3P-g256-EF512.gguf -c 8192 -t $(nproc)
# 3c. server with an OpenAI-compatible API on :8080
./sasori-runtime/bin/llama-server -m Qwen3-14B-TQ3P-g256-EF512.gguf -c 8192 -t $(nproc)
Verified server call (returned 21, correct, in 4.98 s on 8 cores):
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages":[{"role":"user","content":"/no_think How much do 7 pencils cost at 3 reais each? Just the number."}],
"max_tokens":64,"temperature":0}'
If curl is not installed (minimal installs often lack it), the same call with wget:
wget -qO- --header='Content-Type: application/json' \
--post-data='{"messages":[{"role":"user","content":"/no_think How much do 7 pencils cost at 3 reais each? Just the number."}],"max_tokens":64,"temperature":0}' \
http://127.0.0.1:8080/v1/chat/completions
Qwen3 is a reasoning model with thinking on by default: before answering "21" it writes
paragraphs of deliberation. This is not cosmetic — in our clean-machine test, -n 128 without
/no_think burned the entire token budget deliberating and got cut off before ever giving the
answer. With /no_think the same question answered 21 reais immediately. At a few tok/s on a CPU,
always prefix /no_think unless you specifically want the chain of thought.
-DGGML_NATIVE=OFF, so
it does not depend on the build machine's CPU.ldd reports libm.so.6 and libc.so.6 — nothing else. OpenMP and libstdc++ are
linked statically on purpose, so a fresh distro needs no apt install.-c 8192 (weights 12.42 GiB + KV at 160 KiB/token).| context | KV (f16) | total RAM | on a machine with ~17 GB free |
|---|---|---|---|
-c 4096 | 0.62 GiB | ~13.1 GiB | comfortable |
-c 8192 | 1.25 GiB | 13.74 GiB (measured) | recommended |
-c 16384 | 2.50 GiB | ~14.9 GiB | tight |
-c 32768 | 5.00 GiB | ~17.4 GiB | needs --cache-type-k q8_0 --cache-type-v q8_0 |
Speed varies a lot with the machine — do not treat one number as the spec. Measured on two different 8-core AMD hosts: generation 1.7–5.8 tok/s (prompt 6.9–11.6 tok/s). Same nominal config, ~3x apart. Ternary decode is memory-bandwidth-bound, so RAM bandwidth and contention matter more than clock; a dedicated desktop should land at the upper end, a busy shared machine at the lower one. More cores help up to a point (the bottleneck is bandwidth, not compute).
TQ3P is a custom ggml type (id 43; upstream stops at 41), so it needs llama.cpp @6f3a9f3
plus the sasori patch — which is exactly what the bundled binaries are. Two consequences worth
knowing: if you upgrade llama.cpp yourself the model stops loading, and if upstream ever assigns id
43 to a different type, a stock binary could misread the file instead of refusing it. Keep the model
and the bundled runtime together.
-EF512 was quantized with error feedback: 512 calibration sequences are pushed through the
FP16 model to collect the input Hessian H = Σ xxᵀ of every linear layer, and the ternary codes are
then chosen to minimise the error at the layer output (‖(W−Ŵ)x‖_H) instead of the error in the
weights (‖W−Ŵ‖) — the GPTQ idea, applied per trit-plane.
This costs zero extra bits and zero extra latency: the file is identical in size and the CPU
kernel is value-independent (dense maddubs, no zero-skipping), so which trits are zero does not
change the speed. The calibration is a one-off manufacturing cost on a GPU; running the model needs
neither.
bpw = 2K + 16K/g. K=1 (the "1.58-bit" regime) collapses on reasoning in our measurements; K=2 is
model-dependent; K=3 is where quality comes back across model families. So this is not smaller
than 4-bit — Q4_K_M is both smaller and slightly faster at decode on the same machine. The
trade-off here is quality-per-bit with a multiplication-free CPU kernel.
Measured on these exact files, same GPU backend, same 100 GSM8K questions, 5-shot, lm-eval official:
| file | flexible-extract | strict-match |
|---|---|---|
Qwen3-14B-TQ3P-g256.gguf (data-free) | 0.75 ± 0.044 | 0.91 ± 0.029 |
Qwen3-14B-TQ3P-g256-EF512.gguf | 0.87 ± 0.034 | 0.92 ± 0.027 |
Read this carefully, because the headline is misleading on its own. strict-match — did the model
emit a correct #### <answer> — is essentially tied (0.91 vs 0.92): both files solve the same
problems. The 12-point gap is in flexible-extract, which takes the last number in the output and
therefore punishes a model that keeps rambling after answering. So error-feedback mainly buys
output discipline, not reasoning ability. It is still the file to use — it is ≥ on both metrics
and costs nothing — but do not read "+12 pp" as "+12 pp smarter".
Caveats, stated plainly: N=100 (binomial error ≈ ±4 pp), single seed, one task. Not a paired McNemar test — samples were not logged per item, so the flexible-extract gap is significant only under an unpaired test (p ≈ 0.03) while the strict-match gap is noise. For reference, our pipeline previously measured this recipe at 0.755 on GSM8K (N=200) and 0.68 on MATH-500 (N=200, 95.8 % of FP16); the 0.75 above reproduces that independently, on a different backend.