Qwen2.5-14B-Instruct-Pollard

Qwen2.5-14B-Instruct — Pollard

Pollard shrank this model: 29.60 GB (f16) → 3.92 GB — 87% smaller, 7.6× down.

The smallest rung here; larger, higher-fidelity rungs are listed below.

format	this model's size
f16	29.60 GB
Q8_0	~15.69 GB
Q6_K	~12.14 GB
Q4_K_M	~8.58 GB
PollardMix (this repo's IQ1_KT)	3.92 GB

Pollard builds of Qwen/Qwen2.5-14B-Instruct made with Pollard Weights — a ladder of measured-allocation quants (bits placed by per-layer sensitivity, not a uniform crush).

Standard GGUF — runs in stock llama.cpp / ik_llama.cpp, Ollama, LM Studio, except where noted. IQ1_KT needs ik_llama.cpp: its allocation puts ik_llama-only atoms on the tensors it protects. The rest run anywhere.

Model details

Which file should I choose?

Every rung is the same weights, sized to a different RAM budget by the measured allocation. Pick the largest one that fits your machine with room for context:

• ~14 GB RAM / VRAM → Q6_K (12.12 GB). stock atoms — runs anywhere. See Speed below: 12.12 GB does not stay resident on a 16 GB card
• ~11 GB RAM / VRAM → IQ4_XS (8.71 GB). stock atoms — best size/quality point on this ladder
• ~9 GB RAM / VRAM → IQ3_S (7.38 GB). stock atoms
• ~6 GB RAM / VRAM → IQ1_KT (3.92 GB). (ik_llama.cpp) trellis atoms — ik_llama.cpp only

Available files (WikiText-2 raw test, ctx 2048, 145 chunks)

f16 reference PPL 4.9718 ±0.0288.

tok/s measured on an RTX 5070 Ti (16 GB), full GPU offload, Windows/CUDA.

The numbers (WikiText-2 raw, ctx 2048, 145 chunks)

Perplexity with error bars, and the trellis rung against the uniform quants it competes with. Sizes are the published byte counts, in GB:

Q6_K costs 0.4% perplexity against the f16 reference at 41% of its size, and IQ4_XS costs 3.8% at 29%. IQ3_S is the first rung where the cost is visible rather than academic, at +10.6%.

PollardMix beats the uniform 1-bit trellis quant on every metric — PPL −14%, Mean KLD −23%, Median KLD −30%, top-1 +4.4 pts — at +8.5% size, and stays under the 2-bit ceiling. It also passes a chat-coherence gate (explanation, code, reasoning, creative continuation all coherent).

Speed

Decode speed on the same machine, full GPU offload:

Two rows there are worth reading rather than skimming.

IQ1_KT is the smallest file and the slowest measured rung — 53.1 tok/s against IQ3_S's 92.6. Trellis atoms trade decode compute for bytes, so on a GPU that is not memory-starved the small rung loses. Reach for IQ1_KT when the model would not otherwise fit; reach for IQ3_S or IQ4_XS when it does.

Q6_K is listed as unmeasured on purpose. At 12.12 GB it does not stay resident on a 16 GB card once the desktop, the KV cache and the compute buffers are accounted for, and every attempt so far was taken while the GPU was shared — yielding figures between 1.8 and 8.4 tok/s that measure contention, not the model. A number that low reads as a property of the quant, which it is not, so the cell stays empty until the measurement is clean.

Allocation (the surgery)

The trellis rung, tensor by tensor:

Prompt format

ChatML, the Qwen2.5-Instruct template:

<|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant

The template is embedded in the GGUF metadata, so llama.cpp, Ollama and LM Studio apply it for you.

Download a specific file

pip install -U "huggingface_hub[cli]"
hf download PollardWeights/Qwen2.5-14B-Instruct-Pollard \
  --include "Qwen2.5-14B-Instruct-Pollard-IQ1_KT.gguf" --local-dir ./

How to run

IQ1_KT is built on ik_llama-only atoms, so it runs with ik_llama.cpp:

llama-cli    -m Qwen2.5-14B-Instruct-Pollard-IQ1_KT.gguf -ngl 99 -p "Explain why the sky is blue."
llama-server -m Qwen2.5-14B-Instruct-Pollard-IQ1_KT.gguf -ngl 99

For stock llama.cpp, Ollama or LM Studio, use Q6_K instead:

llama-server -hf PollardWeights/Qwen2.5-14B-Instruct-Pollard:Q6_K
llama-cli    -m Qwen2.5-14B-Instruct-Pollard-Q6_K.gguf -ngl 99 -p "Explain why the sky is blue."

imatrix (calibration)

The importance matrix (qwen14b_calib3.imatrix, included) was computed on Calib-3.0: 13.9 MB of mixed prose, code, reasoning and dialogue, 60 chunks at ctx 512, computed CPU-only. Checked disjoint from the WikiText-2 test split used for the perplexity table — none of the 1,723 test lines over 200 characters appears anywhere in the calibration corpus, so the numbers above are not measured on text the allocation was tuned on.

ARM / AVX

llama.cpp repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines — no special file needed, online repacking covers these quants. The old Q4_0_4_4/4_8/8_8 variants are not required.

Errata

• IQ1_KT carries ik_llama-only atoms and needs ik_llama.cpp to run; stock llama.cpp rejects any ggml type above 42 outright. Checked with pollard-ggufcheck, from the files' tensor types rather than their names.
• Measured allocation places bits by per-layer sensitivity under a size budget.
• Single machine; replication invited.

Credits & license

• Base model: Qwen/Qwen2.5-14B-Instruct (Qwen)
• Quantization tooling: llama.cpp (ggml-org)
• Method + tooling: Pollard Weights — measure first, no claim before a number.
• License: apache-2.0, inherited from the base model.

Built with Pollard Weights — frontier models, small hardware, no compromise.