kuclab-hertz-0.6

KucLab Hertz 0.6

A Czech/English STEM + programming assistant built by KucLab on top of google/gemma-4-12B-it. This release recovers and surpasses the STEM accuracy lost in Hertz 0.5, while keeping 0.5's direct/witty personality.

What this is

Hertz 0.6 is a LoRA fine-tune (r=16, merged into the base weights). Unlike 0.3-0.5, most of the training data was not self-distilled from a teacher model this round — the bulk (1994 rows) is a fresh, externally-generated corpus, checked for validity (0 malformed rows, 0 duplicate instructions/outputs, 0 special tokens, 0 identity leaks) before use. It's combined with 606 rows carried forward from Hertz 0.5's corpus (itself the product of three prior training generations), 309 CS↔EN scientific-terminology rows built deterministically from a fixed term list, 8 answer-first formatting examples, and 15 hand-written identity rows.

• Base: google/gemma-4-12B-it (11.95B params, Apache 2.0)
• Method: QLoRA, r=16 / alpha=32, merged to bf16 then quantized
• Context: 262144 tokens natively. We tried extending this via YaRN rope scaling and hit an architecture wall: Gemma-4 stores RoPE settings in a nested rope_parameters structure (separate config per attention type), and rope_scaling is a legacy alias for that same field — writing one clobbers the other and breaks GGUF export. Context stays at the native 262144; a real extension would need a different technique than YaRN on this architecture.
• Training data: 2932 rows total. See the honest development story below — this number is smaller than a first attempt that scored worse.
• Format available: GGUF (q4_k_m, ~7.4GB) for llama.cpp/Ollama, plus the raw LoRA adapter.

Quickstart (Ollama)

Important: ollama pull hf.co/... alone does NOT apply this model's system prompt (identity + personality) — Ollama only fetches the raw GGUF from Hugging Face, it does not read a repo's Modelfile. Without the system prompt, the model falls back to identifying as a generic Gemma model. Use ollama create with the Modelfile below instead — it pulls the weights AND applies the system prompt in one step:

curl -O https://huggingface.co/KucLab/kuclab-hertz-0.6/resolve/main/Modelfile
ollama create kuclab-hertz-0.6 -f Modelfile
ollama run kuclab-hertz-0.6

(The Modelfile's FROM line points at hf.co/KucLab/kuclab-hertz-0.6:Q4_K_M, so this pulls the same GGUF automatically — no separate download needed.)

The honest development story

We're publishing this because it's the kind of thing that usually gets edited out of a release note, and we think it's more useful left in.

Attempt 1 used 2944 rows including 927 terminology rows (three phrasings per term, many of them bare one-word answers) and LoRA rank 32. Result: MMLU-Pro STEM 77.1%, but Czech terminology dropped to 71.4% (from 0.5's 75.7%), with the specifically-targeted EN→CS direction collapsing to 59.2%. Diagnosis: the bare-lookup rows taught the model to recall the 309 training pairs by rote rather than how Czech scientific terms are formed, so it fell apart on the benchmark's held-out quarter.

Attempt 2 fixed that (definitions only, no bare lookups, rank down to 16) but Czech terminology got worse still — 69.4%. Diagnosis: 0.6 was built entirely from a single fresh batch of data with nothing reused from earlier releases. 0.3→0.4→0.5 had each carried forward and re-reinforced the same Czech STEM vocabulary across three training generations; throwing that away for "everything fresh" lost more than 309 new terminology rows could replace in one round.

Attempt 3 (this release) added 606 rows back from Hertz 0.5's corpus alongside the fresh data, restoring some of that cumulative reinforcement. Also, a GGUF export crashed with KeyError: 'full_attention' on the first attempt at YaRN context extension — the root cause (rope_scaling clobbering Gemma-4's nested rope_parameters) is described above and is now guarded against in the export code rather than silently corrupting the config.

Net result: MMLU-Pro STEM ended at 79.2%, Czech terminology at 73.8% — real improvement over 0.5 on STEM, still short of 0.5's terminology peak. We're not aware of a way to have fully matched both without another full iteration cycle, which we didn't have time for before this release's deadline.

Benchmarks

Same prompts, same grading code, same Ollama Q4_K_M quantization, cold, identical corrected methodology throughout (see Hertz 0.5's card for the timeout bug we found and fixed there — this release inherits that fix).

MMLU-Pro STEM (240 held-out questions, this project's own curated subset)

Hertz 0.6 beats the base model by +3.4pp and every prior Hertz release on this benchmark. It also resolved far more answers cleanly: only 18/240 (7.5%) needed the fallback answer-only re-ask, down from 46/240 (19%) in Hertz 0.5 — a direct result of training on answer-first format examples, not a benchmark-harness artifact.

Czech terminology benchmark (206 held-out CS↔EN scientific terms)

Below Hertz 0.5's peak, above Hertz 0.3. See the development story above for why.

Honest status

• ✅ MMLU-Pro STEM: 79.2%, beats base (75.8%) and every prior Hertz release
• ✅ Personality retained from 0.5: direct, witty, no reflexive AI hedging on ordinary topics; genuinely harmful requests still refused
• ✅ Correctly identifies as a KucLab model, no founder named
• ⚠️ Czech terminology (73.8%) is below Hertz 0.5's 75.7% — a real, disclosed regression, not fully recovered despite a dedicated attempt (see development story)
• ⏳ Context extension beyond native 262144 was attempted and found architecturally blocked on this base via YaRN — not solved this release
• ⏳ No tool-calling fine-tuning; no uncensoring pass beyond the personality shift already in 0.5

License

Apache 2.0, inherited from google/gemma-4-12B-it (per Google's official Hugging Face listing).

Credits

• Base model: google/gemma-4-12B-it (Google, Apache 2.0)
• Fine-tuning, dataset construction, and packaging: KucLab