Gemma-3-4b-it-Opusfied

Gemma 3 4B - Voice & Style (Full Finetune / Muon / 12k)

A full fine-tune of Google's Gemma 3 4B IT optimized for specific voice emulation and writing style transfer while maintaining long-context reasoning capabilities.

Unlike standard LoRA adapters, this is a native weight fine-tune trained on an A100 80GB using the Muon optimizer to effectively converge dense layers at a 12k context window.

Model Details

• Base Model: google/gemma-3-4b-it
• Parameter Count: 4.24B
• Training Method: Full Fine-Tune (FFT) - All parameters updated.
• Optimizer: Muon (Momentum Orthogonalized)
• Context Window: 12,288 tokens
• Precision: bfloat16

Datasets

The model was trained on a mix of capability retention and style injection:

1. crownelius/Opus4.6-No-Reasoning-260x: Core voice and conversational style data.
2. crownelius/Opus-4.5-WritingStyle-1000x: Prose and structural writing nuances.
3. crownelius/GLM-5.0-8000x: Logic, math, and reasoning capabilities (Sounds familiar to Opus, and provides CoT reasoning (experimental!)).

Training Configuration

Trained on Modal using a single NVIDIA A100 (80GB).

• Global Batch Size: 16 (Physical Batch 2 + Grad Accum 8)
• Sequence Length: 16,384
• Learning Rate: 5e-5
• Optimizer: Muon
• Gradient Checkpointing: Enabled (Required for 12k context on 80GB VRAM)
• Attention Implementation: Flash Attention 2

Usage

Transformers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "QuietImpostor/Gemma-3-4b-it-Opusfied"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2"
)

messages = [
    {"role": "user", "content": "Explain the concept of entropy in the style of a weary time traveler."},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))

Acknowledgements

• Trained using the Muon optimizer implementation for efficient large-batch training.
• Compute provided by Modal.