Downloads · 30 days
80
30% of all-time downloads
TitanMamo10/qwen25-qat-q3-poc
qwen25-qat-q3-poc is a machine learning model from TitanMamo10. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Quantization-Aware Training artifacts for the custom Q3 GGML grids (Q30G128 / Q31G128 / Q31G64). Training hardware per file is listed below — 0.5B runs fit a GTX 1660 Ti (6GB VRAM, CPU-offloaded optimizer); 1.5B runs…
Downloads · 30 days
80
30% of all-time downloads
All-time downloads
270
Public
Repo size
2.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf2.7 GB · 100%
From the Hugging Face model README
Quantization-Aware Training artifacts for the custom Q3 GGML grids (Q3_0_G128 / Q3_1_G128 / Q3_1_G64). Training hardware per file is listed below — 0.5B runs fit a GTX 1660 Ti (6GB VRAM, CPU-offloaded optimizer); 1.5B runs used a Modal L4 (batch 2, LR 5e-6). See upstream discussion ikawrakow/ik_llama.cpp#2417 for the type proposal.
| File | What | Trained on | PPL (wikitext2-style held-out) |
|---|---|---|---|
stock-f16.gguf | Qwen2.5-0.5B stock F16 reference (949M) | n/a (stock) | 14.74 |
q3distill8-mixed-Q3_1_G64.gguf | 0.5B, LLM-QAT self-distill, mixed grid (embed/lm_head/first+last blocks F16, rest Q3_1_G64), 460M | GTX 1660 Ti 6GB, batch 1, 3300 steps | 21.88 (dg+0.0002 vs own F16 — lossless transfer) |
qwen15_g64_8000-mixed-Q3_1_G64.gguf | Qwen2.5-1.5B, same recipe, 8000 steps, 1.2G | Modal L4, batch 2, LR 5e-6 | 14.10 |
The format/runtime side is done: these files load and run in the
build-qat75 tree (custom Q3 kernels, dp4a MMVQ), greedy generation is
token-identical to F16, transfer degrades +0.0002. The training side is
still cooking: 0.5B sits at 21.88 vs the 16.5 bar; the loop needs 1.5B
scale + more data (100–170k blocks) to close it. PoC model sizes (0.5B,
1.5B) were chosen because they iterate in hours, not weeks (0.5B fits the
local 6GB loop; 1.5B runs on a rented L4).
llama-perplexity -m q3distill8-mixed-Q3_1_G64.gguf -f <your 512-tok chunks> -c 512 -ngl 99
Needs a build with the Q3_1_G64 kernels (see discussion for the patch series once posted).