Downloads · 30 days
266
42% of all-time downloads
MoAI-gemma-4-12B-it-mxfp4
BCCard/MoAI-gemma-4-12B-it-mxfp4
MoAI-gemma-4-12B-it-mxfp4 is a machine learning model from BCCard. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for vllm. The card lists the license as apache-2.0.
google/gemma-4-12B-it quantized to MXFP4 (W4A16) using GPTQ with special techniques that preserve Korean and English performance simultaneously.
All-time downloads
633
Public
Parameters
6.9B
7.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors7.9 GB · 100%
Parameter types
How the weights are stored.
U85.8B · 85%
Base models
Model card
From the Hugging Face model README
Gemma 4 12B IT — MXFP4 Quantized
google/gemma-4-12B-it quantized to MXFP4 (W4A16) using GPTQ with special techniques that preserve Korean and English performance simultaneously.
Model Details
| Property | Value |
|---|---|
| Base model | google/gemma-4-12B-it (BF16, ~24.4 GB) |
| Quantization format | MXFP4 W4A16 (compressed-tensors) |
| Disk size | 7.9 GB (68% reduction from BF16) |
| VRAM (model weights) | ~8 GB |
| Quantized scope | Decoder Linear layers (lm_head, embed_vision, embed_audio kept in BF16) |
| Calibration data | BCCard on-policy dataset (Korean/English split) |
The 12B checkpoint is the
gemma4_unifiedarchitecture variant — its vision stack (embed_vision.*) differs from the 31B'svision_tower, and is kept at original precision.
Serving
vLLM (recommended)
# Standard environment (CUDA 12.9+):
vllm serve gemma-4-12B-it-MXFP4 \
--port 8000 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
# Blackwell with CUDA < 12.9:
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve gemma-4-12B-it-MXFP4 \
--port 8000 \
--linear-backend marlin \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
Blackwell Compatibility
On Blackwell GPUs (SM 12.x) with containers running CUDA < 12.9, FlashInfer JIT compilation fails. The following workarounds are required:
| Setting | Purpose |
|---|---|
VLLM_USE_FLASHINFER_SAMPLER=0 | Disables FlashInfer sampler (uses native PyTorch) |
--linear-backend marlin | Uses pre-compiled MXFP4 GEMM kernels (no JIT) |
Component matrix after workarounds:
| Component | Backend | JIT Required |
|---|---|---|
| MXFP4 GEMM | Marlin (pre-compiled) | No |
| Attention | TRITON_ATTN | No (Triton JIT, not nvcc) |
| Sampling | Native PyTorch | No |
Evaluation
Quality Benchmarks
| Task | MXFP4 (this model) | BF16 baseline | Recovery |
|---|---|---|---|
| HellaSwag (English) | 83.5% | 81.0% | 103.1% |
| KoBEST-HellaSwag | 62.0% | 64.0% | 96.9% |
| KoBEST-COPA | 98.5% | 98.5% | 100.0% |
| KoBEST-BoolQ | 94.0% | 93.0% | 101.1% |
| KoBEST-SentiNeg | 99.0% | 99.5% | 99.5% |
| KoBEST-WiC | 82.0% | 79.0% | 103.8% |
| KMMLU (5 subjects) | 55.5% | 60.5% | 91.7% |
| BC-Finance QA (char-F1) | 0.350 | 0.344 | 101.7% |
| Average recovery | 99.7% | ||
| Korean Generation | 495 tok, rep 1.00, Hangul 58.8% | 492 tok, rep 1.00, Hangul 58.3% | — |
| English Generation | 430 tok, rep 0.99 | 432 tok, rep 0.99 | — |
Korean Knowledge (KMMLU) — Reference Points
Published scores from other Korean(-capable) models, for orientation. These are NOT apples-to-apples: vendors use the full 45-subject KMMLU (or the harder KMMLU-Redux subset) with their own prompting/shot settings, while this model's score is a 5-subject, 200-sample, 0-shot chat-MCQ sample from the harness above.
| Model | Params | Disk | Score | Benchmark (setting/source) |
|---|---|---|---|---|
| A.X 4.0 (SKT) | 72B | ~144 GB | 78.3 | KMMLU, vendor |
| EXAONE 4.0 (LG) | 32B | ~64 GB | 75.2 | KMMLU, as reported in the HyperCLOVA X 32B Think report |
| GPT-4o (OpenAI) | — | — (API) | 72.5 | KMMLU, as reported by SKT A.X |
| HyperCLOVA X 32B Think (NAVER) | 32B | ~64 GB | 71.3 | KMMLU, vendor report |
| A.X 3.1 (SKT) | 34B | ~68 GB | 69.2 | KMMLU, vendor |
| A.X 3.1 Light (SKT) | 7B | ~14 GB | 61.7 | KMMLU, vendor |
| Gemma 4 12B IT (BF16 baseline) | 12B | ~24.4 GB | 60.5 | KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| This model (Gemma 4 12B MXFP4) | 12B | 7.9 GB | 55.5 | KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| Gemma 3 27B IT (Google) | 27B | ~54 GB | 54.0 | KMMLU-Redux, KMMLU-Redux/Pro paper |
| HyperCLOVA X SEED (NAVER) | 3B | ~6.5 GB | 48.5 | KMMLU, vendor |
| Gemma 3 12B IT (Google) | 12B | ~24 GB | 46.7 | KMMLU-Redux, KMMLU-Redux/Pro paper |
Disk sizes for third-party models are approximate BF16 weight footprints (params × 2 bytes); this repo's model and its BF16 baseline are measured on-disk values.
Directional takeaway: at 12B/4-bit this model lands well above the previous Gemma generation's 12B (and its 27B on the harder Redux subset) on Korean knowledge, while purpose-built Korean 32B+/72B models remain ahead — as expected for their size. Against its own BF16 baseline under the identical harness, the model averages 99.7% recovery across all scored tasks, with KMMLU (91.7%) the only drop beyond sampling noise.
Compression Comparison
| Format | Disk Size | vs BF16 | Notes |
|---|---|---|---|
| BF16 (original) | ~24.4 GB | 100% | google/gemma-4-12B-it |
| FP8-dynamic | 13 GB | ~53% | community quantization with the same llm-compressor FP8_DYNAMIC recipe (no official RedHatAI 12B release) |
| This Model (MXFP4) | 7.9 GB | 32% | W4A16 GPTQ + MHR + PSA |
Citation
@misc{gemma4-12b-mxfp4-mhr-psa,
title={Gemma 4 12B IT MXFP4},
author={Taeyoung Lee},
year={2026},
url={https://huggingface.co/BCCard/MoAI-gemma-4-12B-it-mxfp4}
}
License
This model is licensed under the Apache 2.0.
- Library
- vllm
- Type
- gemma4_unified
- License
- apache-2.0
- Languages
- ko, en
- Created
- Jul 12, 2026
- Updated
- Jul 12, 2026