Downloads · 30 days
10.6K
38% of all-time downloads
hiwaifu-research/WaifuGemma4-26b-a4b-v1-i1-GGUF
WaifuGemma4-26b-a4b-v1-i1-GGUF is a text generation model from hiwaifu-research. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
WaifuGemma4-26b-a4b-v1 is a role-play model: Gemma 4 26B-A4B (25.2B total, 3.8B active) post-trained with GRPO against a reward model learned from 1.2M blind votes cast by real users on the HiWaifu Arena. In blind hea…
Downloads · 30 days
10.6K
38% of all-time downloads
All-time downloads
27.9K
Public
Repo size
80.3 GB
Likes
24
Trending 2
Click a slice to open those files.
.gguf80.3 GB · 100%
From the Hugging Face model README
WaifuGemma4-26b-a4b-v1 is a role-play model: Gemma 4 26B-A4B (25.2B total, 3.8B active) post-trained with GRPO against a reward model learned from 1.2M blind votes cast by real users on the HiWaifu Arena. In blind head-to-head battles it splits votes evenly with GLM-5.1 and beats the untuned Gemma 4 26B-A4B-it 59.7% of the time. The full training story, arena results and response comparisons are on the model card.
This repository holds the weighted/imatrix quantizations. Static quants and the vision projector (mmproj) are in WaifuGemma4-26b-a4b-v1-GGUF. Compared with the static quants, the imatrix versions cut the divergence from bf16 roughly in half at 4 bits: i1-Q4_K_M has a mean KLD of 0.069425 against 0.150663 for static Q4_K_M at the same 16.8 GB, and i1-IQ4_XS at 13.9 GB beats static Q4_K_M at 16.8 GB.
| Quant | Size (GB) | Mean KLD vs bf16 | Top-1 agreement | PPL (RP replies) | Notes |
|---|---|---|---|---|---|
| i1-Q4_K_M | 16.8 | 0.0694 | 91.0% | 5.80 | recommended; half the KLD of static Q4_K_M at the same size |
| i1-Q4_K_S | 15.5 | 0.0727 | 90.2% | 5.84 | recommended; slightly smaller |
| i1-IQ4_XS | 13.9 | 0.1015 | 89.3% | 6.11 | recommended for 16 GB cards |
| i1-IQ3_M | 12.4 | 0.1455 | 86.0% | 6.00 | matches static Q4_K_M quality at 12.4 GB |
| i1-IQ3_XXS | 11.3 | 0.3314 | 78.2% | 6.34 | noticeable loss; for 12 GB cards |
| i1-IQ2_M | 10.4 | 0.3997 | 77.4% | 6.82 | largest loss in the set; for 12 GB cards |
| imatrix | 0.06 | – | – | – | importance matrix, for making your own quants |
For reference, the static quants in the other repo:
| Quant | Size (GB) | Mean KLD vs bf16 | Top-1 agreement | PPL (RP replies) | Notes |
|---|---|---|---|---|---|
| Q8_0 | 26.9 | 0.0098 | 96.5% | 5.85 | closest to bf16; 32 GB+ VRAM or CPU RAM |
| Q6_K | 22.6 | 0.0225 | 94.3% | 5.76 | very good; the sweet spot if you have the memory |
| Q5_K_M | 19.1 | 0.0581 | 91.3% | 6.11 | good |
| Q4_K_M | 16.8 | 0.1507 | 85.9% | 6.12 | fast; the imatrix Q4_K_M in the i1 repo is clearly better at the same size |
Any recent llama.cpp works (the GGUFs were made with build 4fea119, September 2026). The model was trained and evaluated in non-thinking mode; pass --reasoning-budget 0 so the template inserts the empty thought channel the model expects.
# llama.cpp server, downloads the file straight from this repo
llama-server -hf hiwaifu-research/WaifuGemma4-26b-a4b-v1-i1-GGUF:Q4_K_M --jinja --reasoning-budget 0 -c 16384 -ngl 99
# then point SillyTavern / any OpenAI-compatible client at http://localhost:8080/v1
Respond in no more than N tokens. to the system prompt to rein it in; it listens.Never speak or act for {{user}}. is the single most useful line.mmproj-WaifuGemma4-26b-a4b-v1-f16.gguf; download it and pass --mmproj if you want image input (or add --no-mmproj to silence the lookup). Image input was untouched by training and has not been evaluated for role-play.The three quality columns come from llama-perplexity --kl-divergence against the bf16 GGUF: mean KL divergence of the quant's next-token distribution from bf16's, the share of tokens where both pick the same top token, and the quant's own perplexity. The text is 20 chunks × 512 tokens of role-play replies written by this model in ten languages, held out from the imatrix calibration set.
One detail matters for anyone reproducing this. Gemma 4 instruction-tuned models, Google's included, only produce calibrated likelihoods inside a model turn; on raw text llama-perplexity reports perplexities in the tens of thousands for every Gemma 4 IT GGUF, which says nothing about the quant. We measured with the model-turn prefix <|channel>thought\n<channel|> inserted after BOS at the start of every chunk, which is what the chat template puts in front of every reply in non-thinking mode. With that prefix the bf16 model scores PPL 5.82 on the held-out replies. The patch is small: in tools/perplexity/perplexity.cpp, right after the tool replaces the first token of each chunk with BOS (in both perplexity() and kl_divergence()), overwrite the next four tokens with ids 100, 45518, 107, 101, and pass parse_special=true to common_tokenize so chat-formatted files tokenize correctly. Without it, compare quants by KL divergence only and ignore the absolute perplexity.
convert_hf_to_gguf.py --outtype bf16, then llama-quantize from the bf16 GGUF with --imatrix.tokenizer_config.json is in the transformers-5 format (extra_special_tokens is a list). llama.cpp pins transformers 4.57.6, which expects a dict and fails with AttributeError: 'list' object has no attribute 'keys' in _set_model_specific_special_tokens. Either install transformers>=5 in the conversion environment or change that field to {"video_token": "<|video|>"} before converting; both produce identical vocabularies. The same applies to Google's own Gemma 4 checkpoints.--parse-special) of 200 real role-play conversations from the HiWaifu Arena, formatted with the Gemma 4 chat template exactly as the model sees them at inference: system prompt, history, and the model's reply behind the non-thinking prefix. Languages: Spanish, Russian, English, Portuguese, Indonesian, Arabic, Thai, French, Ukrainian, German. The imatrix file is included so you can make your own quants.Apache 2.0, as inherited from the base model.