Downloads · 30 days
0
Esther125/gemma-4-31b-it-zhtw-cpt-v2-10k
gemma-4-31b-it-zhtw-cpt-v2-10k is a machine learning model from Esther125. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
This repository stores three layer-wise LoRA adapters trained from google/gemma-4-31B-it with continued pretraining (CPT) on 10k Taiwan Traditional Chinese text examples.
Downloads · 30 days
0
Access
Public
Updated Jul 25, 2026
Repo size
423 MB
Likes
0
Public
Click a slice to open those files.
.safetensors391 MB · 80%
From the Hugging Face model README
This repository stores three layer-wise LoRA adapters trained from google/gemma-4-31B-it with continued pretraining (CPT) on 10k Taiwan Traditional Chinese text examples.
Unlike the V1 instruction-tuning adapters, V2 uses plain text and causal language-modeling loss. Its goal is to test whether Taiwan-sourced dictionary and local text can change the model's natural preference for Taiwan terminology.
{"text": "..."}| Source | Examples |
|---|---|
| MOE Concised Mandarin Chinese Dictionary definitions | 4,037 |
| MOE Concised Dictionary example sentences | 4,271 |
| MOE Dictionary of Chinese Idioms entries and examples | 1,558 |
| Taiwan travel text | 134 |
| Total | 10,000 |
The corpus is dominated by short dictionary-style text. It is useful for terminology adaptation, but it is not representative of complete Taiwan conversations or general instruction-following data.
google/gemma-4-31B-ittext + eosq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj| Folder | Strategy | LoRA layers | Train loss | Runtime |
|---|---|---|---|---|
strategy_a_tw_cpt_v2_10k/ | Overall critical layers | 12-19 | 4.6808 | 86.5 min |
strategy_b_tw_cpt_v2_10k/ | Critical + high Traditional/Simplified contrast layers | 8-10, 12-19, 58 | 3.3085 | 105.7 min |
strategy_c_tw_cpt_v2_10k/ | Critical + moderate layers | 1-24, 55-58 | 2.9838 | 119.3 min |
The strategies were derived from activation-based layer probing:
Each strategy folder is a standalone PEFT adapter. Pass the repository ID and strategy folder separately:
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model_id = "google/gemma-4-31B-it"
adapter_repo_id = "Esther125/gemma-4-31b-it-zhtw-cpt-v2-10k"
adapter_subfolder = "strategy_b_tw_cpt_v2_10k"
tokenizer = AutoTokenizer.from_pretrained(
adapter_repo_id,
subfolder=adapter_subfolder,
)
model = AutoModelForCausalLM.from_pretrained(
base_model_id,
device_map="auto",
)
model = PeftModel.from_pretrained(
model,
adapter_repo_id,
subfolder=adapter_subfolder,
)
A 200-prompt benchmark was built from 100 manually reviewed Taiwan/Mainland China terminology mappings. Each mapping has one controlled lexical prompt and one natural-generation prompt. Prompts do not request a specific regional variety.
| Model | Controlled TW preference | Natural TW preference | Overall TW preference | Trigger rate |
|---|---|---|---|---|
| Base | 90.6% | 88.3% | 89.5% | 81.0% |
| V2 Strategy A | 91.9% | 87.2% | 89.6% | 82.0% |
| V2 Strategy B | 94.4% | 92.5% | 93.5% | 84.5% |
| V2 Strategy C | 95.5% | 92.6% | 94.1% | 85.0% |
Strategies B and C improved preference for the benchmark's targeted Taiwan terms by 4.0 and 4.6 percentage points over the base model. Strategy B used 12 layers and finished within 0.6 percentage points of the 28-layer Strategy C.
This benchmark measures preference for 100 reviewed terminology mappings. It should not be interpreted as a complete measure of Taiwan Chinese generation quality.
Small ARC-Challenge, BBH Boolean Expressions, and GSM8K subsets were used as regression checks.
| Model | ARC | BBH | GSM8K | Macro accuracy |
|---|---|---|---|---|
| Base | 0.9333 | 1.0000 | 0.9333 | 0.9556 |
| V2 Strategy A | 0.9333 | 1.0000 | 0.8667 | 0.9333 |
| V2 Strategy B | 0.9333 | 1.0000 | 0.9333 | 0.9556 |
| V2 Strategy C | 0.9333 | 1.0000 | 0.8667 | 0.9333 |
These are small regression checks, not full benchmark results. Strategy B was the only V2 adapter that matched the base model's macro accuracy.
Strategy B is the most balanced research checkpoint: