Downloads · 30 days
71
12% of all-time downloads
julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer
occitan-gemma-4-12b-it-rslora-qat-sfttrainer is a text generation model from julienp79. Use it when you need the model to write or continue text. It is set up for peft.
This repository contains a fine-tuned version of Google's Gemma 4 12B Instruct with a QAT-aware base (gemma-4-12B-it-qat-q40-unquantized), optimized for the Occitan language using RS-LoRA (Rank-Stabilized Low-Rank Ada…
Downloads · 30 days
71
12% of all-time downloads
All-time downloads
587
Public
Parameters
12B
81.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.gguf57.3 GB · 70%
From the Hugging Face model README
This repository contains a fine-tuned version of Google's Gemma 4 12B Instruct with a QAT-aware base (gemma-4-12B-it-qat-q4_0-unquantized), optimized for the Occitan language using RS-LoRA (Rank-Stabilized Low-Rank Adaptation) via SFTTrainer.
The QAT (Quantization-Aware Training) base model was originally quantized to Q4_0 and then dequantized back to bf16, meaning the weights carry QAT-aware characteristics that may improve robustness to post-training quantization.
transformers)adapter/ Folder: The raw RS-LoRA adapter files (for use with PEFT)gguf/ Folder: Quantized GGUF versions for local inference via llama.cpp, LM Studio, Ollama, etc.from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [
{"role": "user", "content": "Escrivètz un cort paragraf en occitan sus la lenga occitana e son importància."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = AutoModelForCausalLM.from_pretrained(
"google/gemma-4-12B-it-qat-q4_0-unquantized",
device_map="auto",
torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12B-it-qat-q4_0-unquantized")
model = PeftModel.from_pretrained(base_model, "julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer", subfolder="adapter")
model.eval()
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "Escrivètz un cort paragraf en occitan."}],
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
llama-cli -hf julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer:Q4_K_M \
-p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
-n 256 -e --temp 0.7
Or via the GGUF file directly:
llama-cli -m gguf/occitan-gemma-4-12b-it-rslora-qat-sfttrainer-Q4_K_M.gguf \
-p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
-n 256 -e --temp 0.7
| Parameter | Value |
|---|---|
| Adapter Type | RS-LoRA (use_rslora=True) |
| Base Model | google/gemma-4-12B-it-qat-q4_0-unquantized (QAT-aware base) |
| LoRA Rank (r) | 32 |
| LoRA Alpha | 32 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Dropout | 0 |
| Bias | None |
| Training Method | SFT (trl.SFTTrainer) |
The model was trained on raw Occitan text from four categories:
All texts were chunked to 384-token blocks (causal language modeling, no chat templating applied during training).
| Parameter | Value |
|---|---|
| Block Size | 384 tokens |
| Learning Rate | 5e-5 (cosine schedule) |
| Optimizer | Paged AdamW 8-bit |
| Max Gradient Norm | 1.0 |
| Warmup Steps | 400 |
| Weight Decay | 0.01 |
| Epochs | 5 |
| Gradient Accumulation Steps | 8 |
| Per-device Batch Size | 1 |
| Total Steps | ~2535 |
| Final Loss | ~1.80 |
| Total Tokens Processed | ~7.8M |
BitsAndBytes) — matches QAT baseuse_reentrant=True)Available in gguf/:
| Quant | Size | Description |
|---|---|---|
f16 | ~23 GB | Lossless float16 |
Q8_0 | ~12 GB | Near-lossless 8-bit |
Q5_K_M | ~8 GB | High quality 5-bit |
Q4_K_M | ~6.9 GB | Recommended 4-bit (best quality/size) |
Q2_K | ~4.5 GB | Aggressive 2-bit (constrained hardware) |
This model was trained with SFT (Supervised Fine-Tuning) using SFTTrainer on raw Occitan text. No chat templating was applied during training — the model learns language structure via causal language modeling on chunked text. The RS-LoRA variant helps stabilise training at higher ranks by scaling the LoRA update by alpha / sqrt(r) instead of alpha / r.
The QAT base model (gemma-4-12B-it-qat-q4_0-unquantized) was quantized to Q4_0 and dequantized back to bf16 before training, meaning the base weights carry quantization-aware characteristics. Combined with fp4 training quantization, this approach aims to produce a model that is more robust to post-training quantization (e.g., GGUF Q4_K_M or Q2_K).
@software{geyronneau2024trl,
title = {{TRL: Transformers Reinforcement Learning}},
author = {Geyronneau, Quentin and von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
license = {Apache-2.0},
url = {https://github.com/huggingface/trl},
year = {2024}
}