tf2-12b-w8a8

🦊 klusai/tf2-12b-w8a8 — EN→RO Translator (W8A8 via LLM Compressor)

Gemma 3 12B (Instruct), LoRA-tuned on 15k English→Romanian fable pairs (synthetic, GPT-o3), then compressed to W8A8 (INT8 weights & activations) using LLM Compressor by vLLM for fast, memory-efficient inference.

Focus: faithful, fluent EN→RO translations of short moral fables, preserving tone/structure and proper Romanian diacritics.

---

✨ What’s in this build

• Training: Same LoRA fine-tune as [tf2-12b-gguf](https://huggingface.co/klusai/tf2-12b-gguf) (15k EN→RO fables, GPT-o3).
• Compression: W8A8 (INT8 weights + INT8 activations) with LLM Compressor; channel-wise weight quant + dynamic per-token activation quant. ([VLLM Documentation][1])
• Format: Published for Transformers with compressed tensors metadata; runs great on vLLM server. ([Hugging Face][2], [VLLM Documentation][3])
• Hardware notes: INT8 execution supported on modern NVIDIA GPUs (compute capability ≥ 7.5; e.g., Turing/Ampere/Ada/Hopper). ([BookStack][4])

---

🚀 Inference

vLLM

pip install vllm

vllm serve tf2-12b-w8a8/ --dtype bfloat16 --tensor-parallel-size 1 --trust-remote-code --port 8000 --host 0.0.0.0

Then call it with your OpenAI Client (set base_url to your server).

---

✅ Intended Use

• EN→RO machine translation of narrative prose (fables), educational localization, style-preserving story translation.

⚠️ Limitations

• Domain-specific (fables); not tuned for legal/medical slang or highly technical text.
• Synthetic training data may encode stylistic biases; human review recommended for production.

---

🔑 License

• This repository: MIT
• Use must also comply with the Gemma base model license.

---

📝 Changelog

• v1.0 (W8A8): LoRA-merged, quantized with LLM Compressor to W8A8; uploaded with compressed-tensors metadata for Transformers & vLLM. ([VLLM Documentation][5])