gemma-4-26bFT-LuxAlpha_code-MoE

Gemma 4 26B-A4B MoE: Empirical Code SFT on AMD Instinct MI300X

This repository provides a Parameter-Efficient Fine-Tuned (PEFT/LoRA) adapter for Google DeepMind's Gemma 4 26B-A4B Mixture-of-Experts (MoE) model.

It represents an empirical case study exploring unquantized 16-bit bfloat16 LoRA fine-tuning on an AMD Instinct MI300X (192GB HBM3) accelerator across 15,200 instruction pairs, followed by rigorous multi-benchmark execution testing.

---

1. Verified Empirical Benchmark Evaluation

Both the Base Gemma 4 26B-A4B-it model and this Fine-Tuned Checkpoint were evaluated head-to-head under identical sandbox conditions:

Key Empirical Findings

1. Zero Catastrophic Forgetting: The fine-tuning process preserved 100% of the base model's mathematical problem-solving ability on GSM8K.
2. Style & Idiom Adaptation: Token diff analysis revealed that the adapter shifted generation toward real-world open-source GitHub phrasing, structural validation, and robust Pydantic retry loops.

---

2. Hardware & Training Configuration

• Accelerator: 1x AMD Instinct MI300X VF (191.69 GB VRAM, 5.3 TB/s HBM3 memory bandwidth)
• Software Stack: AMD ROCm 7.x, PyTorch 2.6.0+rocm6.1, TRL, Hugging Face PEFT
• Precision: Native 16-bit BF16 (Zero 4-bit quantization degradation)
• LoRA Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Gating Network / Router: Frozen (prevents routing collapse)
• LoRA Hyperparameters: Rank r=32, Alpha=64, Dropout=0.0
• Batch Sizing: Micro-batch 4, Gradient Accumulation 8 (Effective Batch Size = 32)
• Steps Completed: 475 optimization steps (~15,200 training instances)

---

3. Training Datasets

1. ise-uiuc/Magicoder-OSS-Instruct-75K: Program synthesis grounded in open-source GitHub code.
2. m-a-p/CodeFeedback-Filtered-Instruction: Multi-turn code refactoring and bug fixing.

---

4. How to Load and Use This Model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_id = "google/gemma-4-26B-A4B-it"
adapter_id = "lshar/gemma-4-26bFT-LuxAlpha_code-MoE"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "<start_of_turn>user\\nWrite a FastAPI endpoint with Pydantic validation.<end_of_turn>\\n<start_of_turn>model\\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=400, temperature=0.2, do_sample=True)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))