Downloads · 30 days
14
2% of all-time downloads
Kh0rd/gemma-4-12B-it-qat-assistant-q8_0-gguf
gemma-4-12B-it-qat-assistant-q8_0-gguf is a text generation model from Kh0rd. Use it when you need the model to write or continue text. The card lists the license as gemma.
GGUF Q80 conversion of google/gemma-4-12B-it-qat-q40-unquantized-assistant. Intended for use as a draft model with llama-server's speculative decoding (--spec-type draft-mtp) to accelerate the main model's generation…
Downloads · 30 days
14
2% of all-time downloads
All-time downloads
565
Public
Repo size
465 MB
Likes
0
Public
Click a slice to open those files.
.gguf465 MB · 100%
From the Hugging Face model README
GGUF Q8_0 conversion of google/gemma-4-12B-it-qat-q4_0-unquantized-assistant.
Intended for use as a draft model with llama-server's speculative decoding (--spec-type draft-mtp) to accelerate the main model's generation speed.
google/gemma-4-12B-it-qat-q4_0-unquantized-assistant を GGUF Q8_0 に変換したドラフトモデルです。
llama-server の投機的デコード(--spec-type draft-mtp)でメインモデルの生成速度を向上させるために使います。
| File | Size | Description |
|---|---|---|
assistant_q8_0.gguf | 444 MB | Q8_0 quantized / Q8_0 量子化 |
Use together with the main model google/gemma-4-12B-it-qat-q4_0-gguf.
メインモデル google/gemma-4-12B-it-qat-q4_0-gguf と組み合わせて使用します。
llama-server \
-m gemma-4-12b-it-qat-q4_0.gguf \
-md assistant_q8_0.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--batch-size 512 \
--ubatch-size 512 \
--flash-attn on \
-ngl 99 \
-c 32768 \
--port 8081 \
--jinja
| Setup / 構成 | Speed / 生成速度 |
|---|---|
| Main model only / メインモデル単体 | ~37 tok/s |
| With draft-mtp / draft-mtp 併用 | ~62 tok/s (+65%) |
Tested on: Windows 11 / CUDA 12.4 / llama-server b9553
google/gemma-4-12B-it-qat-q4_0-unquantized-assistant (safetensors)
→ convert_hf_to_gguf.py (BF16 GGUF)
→ llama-quantize (Q8_0)
gemma-4-12b-it-qat-q4_0 (QAT-based pair).gemma-4-12b-it-qat-q4_0 である必要があります(QATベース同士での組み合わせ)。--batch-size 512 is required to prevent OOM caused by Gemma 4's large vocabulary size.--batch-size 512 は必須です(Gemma 4 の vocab サイズに起因する OOM 防止)。This model is subject to the Gemma Terms of Use.
Copyright of the original model belongs to Google LLC.
このモデルは Gemma Terms of Use に従います。
元モデルの著作権は Google LLC に帰属します。