Downloads · 30 days
0
xbill9/gemma-4-26B-A4B-it-inferentia2-int8
gemma-4-26B-A4B-it-inferentia2-int8 is a text generation model from xbill9. Use it when you need the model to write or continue text. It is set up for torch-neuronx. The card lists the license as apache-2.0.
An int8-quantized-expert build of google/gemma-4-26B-A4B-it for inf2.24xlarge (TP=8). The 128 MoE experts are quantized to per-channel symmetric int8; every other weight stays bf16. Greedy decode is token-for-token id…
Downloads · 30 days
0
Access
Public
Updated Jul 16, 2026
Repo size
93.5 GB
Likes
0
Public
Click a slice to open those files.
.safetensors51.6 GB · 55%
From the Hugging Face model README
An int8-quantized-expert build of google/gemma-4-26B-A4B-it
for inf2.24xlarge (TP=8). The 128 MoE experts are quantized to per-channel symmetric int8; every
other weight stays bf16. Greedy decode is token-for-token identical to the bf16 / CPU fp32 reference
(SEQ_MATCH True, "The capital of France is Paris.") — int8 per-channel loses nothing here.
Why this variant: the compiled model is 41.8 GB vs 64.6 GB for the bf16 build (the experts are
~93% of the weight; int8 halves them) — a smaller HBM footprint and faster load/serve on the same box,
at bf16-identical quality. See the bf16 build
xbill9/gemma-4-26B-A4B-it-inferentia2
for the full recipe write-up.
Note on smaller boxes: int8 experts save ~22.8 GB but the model is still ~3–4 GB over a 16 GB NeuronCore at TP=2, so this build targets the 24xlarge (12 cores), not the 2-core
inf2.xlarge/inf2.8xlarge. Fitting a 2-core box would need fp4 experts (a further, accuracy-risky step).
Reuses the bf16 MoE recipe (tp_mb_moe.py): all-experts-dense compute, SPMDRank scatter for the
router weighting, mixed-attention shard/replicate. The only change (tp_mb_moe_int8.py) swaps the two
expert linears for QuantizedColumnParallel/QuantizedRowParallel (INT8, per-channel-symmetric)
and quantizes the fused expert weights in checkpoint_loader. Two NxD autograd quirks were fixed to
compile: .clone() the down output, and monkeypatch scale_dequantize to be out-of-place.
| file | what |
|---|---|
mb_26b_int8_tp8.pt | Compiled TP=8 MoE model, int8 experts (~42 GB), KV 256/64 |
real-gemma4-26B-A4B-it/ | Google's weights + tokenizer + chat_template.jinja |
tp_mb_moe_int8.py | The int8 recipe (QuantizedColumnParallel/RowParallel + q8 in checkpoint_loader) |
optb_server_int8.py | HTTP server (OpenAI-compatible + /generate, /metrics, streaming) |
Dockerfile.int8, entrypoint.int8.sh | Thin image that pulls these artifacts at start and serves |
MODEL_DIR=/data/real-gemma4-26B-A4B-it MB_LOAD=/data/mb_26b_int8_tp8.pt \
TP_DEGREE=8 KV_MAX=256 KV_BUCKET=64 python optb_server_int8.py
curl -s localhost:8080/generate -d '{"prompt":"What is the capital of France?"}'
# -> {"response":"The capital of France is Paris.", ...}
Docker: docker run -d --device /dev/neuron0 ... --device /dev/neuron5 --ipc=host -v gemma26b8:/data -p 8080:8080 xbill9/gemma4-optb-26b-int8:latest
Neuron SDK 2.23 · torch-neuronx 2.8.0 · neuronx-distributed 0.17.26814 · transformers 5.13.0.
License follows the upstream Gemma weights' terms; see google/gemma-4-26B-A4B-it.