Tinfield-1-Mini-GGUF

Tinfield 1 Mini

Tinfield 1 Mini is the smaller build of Tinfield 1: 330 GB down to 61 GB, with only 34.7 GiB resident on the GPU. Same recipe as Compact except the expert down projections, which drop to 2 bits.

Two-bit experts usually wreck a model. llama.cpp's stock Q2_0 leaves 51.9% weighted error on these tensors. This build gets that to 11.7%, and mean KL against Q8 lands at 0.340 against Compact's 0.287.

Range Before Representation

llama.cpp's stock Q2_0 takes each block's maximum as its scale, which zeroes most weights and ignores the importance matrix. On these down projections that leaves 51.9% weighted error.

This build searches each block's range instead: 17 clip ratios against both signs, weighted by the importance matrix, keeping whichever minimises the error. That takes the weighted error to 11.7%. The file is laid out by llama.cpp's own quantizer and the data is then rewritten in place, so it loads in stock llama.cpp with no patches.

The encoder now lives inside llama-quantize as a patch against llama.cpp 972d2313b: see kernels/ in this repo, which also carries the IQ2_NL type and a cross-check against the Python encoders.

Running it

llama.cpp must be at commit 972d2313 or later. This architecture is not in older builds, and they fail to load the file rather than warning about it. If the model will not open, check this first.

The six parts are a split GGUF: point llama.cpp at the first part and it finds the rest.

llama-server -m tinfield-1-mini-00001-of-00006.gguf --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20 \
  -fa on -ctk q8_0 -ctv q4_0 \
  -ngl 999 -c 131072

--jinja is required or the chat template is ignored and tool calling breaks. The sampler values are Qwen's own, from generation_config.json. The KV flags cost 9,984 bytes per token of context instead of 24,576, and that cost is what the KL number above accounts for.

Skip n-gram self-speculation. On agent traffic it accepted only 16 to 25% of drafted tokens, which costs more verification work than it saves. Output is identical either way.

Fitting it on your hardware

Experts are 30.0 GiB (0.63 GiB per layer), the spine is 4.8 GiB, and the engram table is 26.8 GiB and always memory-mapped on the CPU side, since a token reads only a handful of its rows.

On the NVMe tier, do not pass --mlock or --no-mmap. The OS cache is doing the work: about 16 GB of expert cache catches roughly 80% of routings.

On macOS, 34.7 GiB on the GPU side exceeds the default wired limit once context is added. Raise it before launching:

sudo sysctl iogpu.wired_limit_mb=56000

launch_presets.sh in this repo wraps all four tiers.

Worth knowing

Reasoning is preserved by default. The chat template keeps prior turns' thinking in context and llama.cpp enables that automatically, so on long agent runs context fills faster than you expect. Pass --no-reasoning-preserve to turn it off.

-c is total context, not per slot. With -np 8 -c 16384 each slot gets 2048 tokens and requests truncate without saying so. Multiply: -np 5 -c 655360 gives five slots of 131072.

First load takes about 35 seconds from fast local storage, longer over a network mount. It is not hung.

Quality

Mean KL divergence against the Q8 build is 0.340, against 0.287 for Compact, measured on 22 held-out session slices at 4,096 context.

On a matched run over five hard Terminal-Bench 2.1 tasks this build and Compact agreed wherever both reached a result. That is a smoke test on two comparable tasks, not a benchmark.

The benchmark scores on the parent model are for the BF16 weights. This build has not been evaluated on Terminal-Bench 4.0 or DeepSWE.

License

Qwen Community License 1.0, following the base model.