Grug-12B-GGUF

This is quantized version of kai-os/Grug-12B created using llama.cpp

Grug 12B is a compact-reasoning fine-tune of google/gemma-4-12B-it. It was trained to keep the useful information from a reasoning trace while making the trace shorter, denser, and less verbose.

This repository is published as merged Transformers/safetensors model weights. It was trained with QLoRA, then merged into the base model for release.

What Changed

The training target is a terse internal-reasoning style: short high-density steps, fewer filler words, and explicit preservation of key constraints, branching decisions, invariants, edge cases, and final-answer checks.

The goal is lower reasoning-token usage relative to the base model while preserving answer quality. It is not meant to hide uncertainty or remove needed reasoning.

Training Data

The data pipeline started from a recent, filtered reasoning pool and converted verbose traces into compact traces before SFT packing.

Source gate:

• Run date: June 30, 2026.
• Default freshness cutoff: 45 days. Sources older than May 16, 2026 were rejected unless manually allowed.
• Allowed train licenses: MIT, Apache-2.0, CC-BY-4.0, CC0-1.0.
• Hard reject terms included OpenAI, ChatGPT, GPT-5, Claude, Anthropic, Opus, Sonnet, and Gemini.
• Soft-risk sources marked as synthetic/distill were manually reviewed or rejected depending on provenance and license.

Final verified source mix:

Row counts:

• Normalized recent reasoning pool: 8,680 rows.
• Selected verbose reasoning set: 6,144 rows.
• Compact raw transform output: 6,144 rows.
• Verified compact rows: 5,740 rows.
• Rejected compact rows: 404 rows.
• Packed SFT split: 5,166 train / 287 validation / 287 test.

The compact reasoning transform was generated with cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit served by vLLM. Rows were checked for compression ratio, answer preservation, and obvious loss of critical reasoning information before training.

Training Procedure

Training was completion-only SFT: prompt tokens were masked with -100, and only the assistant completion was trained.

Core settings:

• Base model: google/gemma-4-12B-it.
• Method: QLoRA / PEFT LoRA, merged into full model weights for upload.
• Quantization during training: 4-bit NF4 with BF16 compute.
• Max sequence length: 6,144.
• LoRA rank: 16.
• LoRA alpha: 32.
• LoRA dropout: 0.05.
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
• Batch size: 1.
• Gradient accumulation: 16.
• Learning rate: 8e-5.
• Max steps: 100.
• Eval steps: 50.
• Save steps: 50.
• Train runtime: about 35 minutes 20 seconds on one A100.
• Final eval loss: 0.8895.

No train or validation rows were skipped in the final run.

Local Evaluation

Small local EOS-only math proxy eval, no generation token cap:

This is a small proxy eval, not a broad benchmark. Treat it as a smoke test showing the intended token-efficiency direction, then run your own benchmark.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kai-os/Grug-12B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

messages = [
    {"role": "user", "content": "If a shirt is $80 and goes 25% off, what is the sale price?"}
]
inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(inputs, do_sample=False, max_new_tokens=512)

print(tokenizer.decode(output[0], skip_special_tokens=True))

For token-efficiency tests, compare against the base model with the same prompt, same decoding settings, and no artificial token cap unless your deployment requires one.

Limitations

• This is an experimental fine-tune.
• It may over-compress reasoning on tasks that need longer derivations.
• It inherits the base model's limitations and safety behavior.
• The reported eval is small and local.
• The dataset includes synthetic and distilled reasoning traces from the listed open datasets; review source licenses and provenance before using this in commercial or sensitive settings.

Acknowledgements

Thanks to Lambda, the inference provider, for compute credits that supported the dataset work, training, and evaluation.