gemma-4-E2B-mini

Gemma 4 E2B Mini — text-only, vocabulary pruned to English + Vietnamese

A compact, vocabulary-pruned version of Gemma 4 E2B (text-only). The 262,144-token multilingual vocabulary is reduced to the 69,246 tokens used by English + Vietnamese, removing ~44% of the parameters with no measured quality loss on EN/VI. Built for efficient on-device finetuning and serving on a 12 GB GPU.

Built on Google's Gemma 4 (via principled-intelligence/gemma-4-E2B-it-text-only, the text-only checkpoint). Use is governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy — https://ai.google.dev/gemma/terms . "Gemma is provided under and subject to the Gemma Terms of Use." This derivative keeps the Gemma name and license as required.

What changed vs. the base

59% of the base is vocab-dependent (token embeddings + Per-Layer Embeddings); these are not quantized during QLoRA, so pruning them is the highest-leverage size/VRAM win on small GPUs. It also raises the max QLoRA training context on a 12 GB card from ~1024 to ~7168 tokens.

How it was made

1. Keep-set = tokens used by an EN+VI corpus (VI+EN Wikipedia) + all 256 byte-fallback tokens + special tokens + every single EN/VI character token (69,246 total).
2. The BPE tokenizer is pruned (merges filtered so every kept merge stays applicable; ids remapped contiguously); byte-fallback remains, so any character is still encodable.
3. embed_tokens, embed_tokens_per_layer (PLE), and the tied lm_head are row-sliced to the keep-set. The original rows are kept verbatim, so EN/VI behaviour is unchanged — only the ability to emit other-language tokens is removed.

Usage (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
ID = "thanglq150188/gemma-4-E2B-mini"
tok = AutoTokenizer.from_pretrained(ID)
m = AutoModelForCausalLM.from_pretrained(ID, dtype=torch.bfloat16, device_map="cuda")
msgs = [{"role": "user", "content": "Thủ đô của Việt Nam là thành phố nào?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True,
        return_tensors="pt", return_dict=True).to("cuda")
print(tok.decode(m.generate(**ids, max_new_tokens=64)[0, ids["input_ids"].shape[1]:],
                 skip_special_tokens=True))

Chat format is Gemma 4's <|turn> / <turn|> (not Gemma 2/3's <start_of_turn>).

Usage (llama.cpp / llama-server)

# gemma4-envi-pruned-q8_0.gguf and gemma4_chat.jinja are included in this repo
llama-server -m gemma4-envi-pruned-q8_0.gguf -ngl 99 -fa on \
  --parallel 8 --ctx-size 65536 --cont-batching \
  --chat-template-file gemma4_chat.jinja

~260 tok/s aggregate at 10 concurrent requests, ~3.0 GB VRAM (8×8192 ctx) on an RTX 3060. The model stays resident (no idle unload).

vsfc-sentiment-lora/ — demo finetune (optional)

QLoRA adapters from finetuning this model on UIT-VSFC Vietnamese sentiment (4k examples). Lifts accuracy 0.68 → 0.95 on the 300-example test set. Load with peft.PeftModel.from_pretrained(base, "thanglq150188/gemma-4-E2B-mini", subfolder="vsfc-sentiment-lora"). Error analysis: 0 of 16 mistakes involved a pruned-away token — pruning is not the cause.

Limitations

• English + Vietnamese only by design. Other languages/scripts now fragment into byte-fallback tokens and are not supported (that is the point — the vocab was trimmed).
• Inherits the capabilities and limitations of the base Gemma 4 E2B model.