gemma-4-12b-it-qat-4bit-frontierdistill-v2

gemma-4-12b-it-qat-4bit-frontierdistill-v2

This is a further post-trained, optimized iteration of True2456/gemma-4-12b-it-qat-4bit-frontierdistill.

This model resolves a critical agentic failure pattern observed in V1: premature turn-stopping, where the agent announces its plan/intent in text but finishes its turn before emitting the corresponding tool call. It has been post-trained with targeted ORPO preference pairs, fused with a Cybersecurity LoRA adapter, and quantized to a flat 4-bit representation.

Trained, fused, and quantized on Apple Silicon via mlx-vlm.

Key Improvements in V2

• ORPO stopping correction: Post-trained on 2,000 ORPO (Odds Ratio Preference Optimization) preference pairs where the chosen response includes the tool call and the rejected response truncates right after the text description.
• Cybersecurity Enhancement: Fused with a custom cybersecurity LoRA adapter for enhanced domain-specific reasoning and safety capabilities.
• Quantization: Quantized to flat 4-bit representation for efficient, fast local inference.

Benchmarks & Evaluation

Note: HumanEval performance improves by +2.5 percentage points compared to V1 (+LoRA)'s score of 89.0%, demonstrating the benefit of targeted ORPO alignment.

Usage

You can run this model locally on Apple Silicon using:

pip install mlx-vlm
python -m mlx_vlm.generate \
    --model True2456/gemma-4-12b-it-qat-4bit-frontierdistill-v2 \
    --prompt "Your prompt here"