gemma4-ushistory-apush-v2-GGUF

gemma4-ushistory-apush-v2-GGUF

GGUF quantizations of ambrosfitz/gemma4-ushistory-apush-v2 (epoch 3, merged 16-bit), for local inference via llama.cpp / LM Studio.

Why epoch 3, not epoch 2

Per-epoch intra-set duplicate rate during training (HF Transformers, from the source repo's TRAIN_REPORT.md):

Epoch 2 scored better on greedy duplicate rate (0.100 vs 0.150); temp 0.7 was tied (0.067). Epoch 3 was the chosen conversion target per this project's handoff, not because it measured better on this specific metric. The epoch-2 adapter remains available in the source repo (adapter/epoch2/, with full optimizer/scheduler state) if a re-convert from that checkpoint is ever wanted.

Required generation recipe

This model was trained and evaluated with a specific chat template and stop-token setup, inherited from the same lineage as gemma4-ushistory-sft-parityfix-GGUF. Getting either wrong produces malformed or garbage output.

• Chat template: gemma-4, non-thinking mode. The correct jinja template (verified to contain the strip_thinking macro and emit <turn|> correctly) is embedded in this GGUF's own metadata (tokenizer.chat_template) — confirmed present and intact via gguf.GGUFReader directly on the conversion pod. Do not override it with a runtime's built-in "gemma" preset — llama.cpp ships a built-in template named gemma for the classic Gemma 1/2/3 format (<start_of_turn>/<end_of_turn>), incompatible with this checkpoint's Gemma-4 format. Leave the chat template on default (LM Studio: don't override it; llama.cpp: don't pass --chat-template).
• Stop token: <turn|> (vocab id 106, confirmed present in the GGUF's vocab at that id). This is not encoded as an EOS/EOG token in GGUF metadata — tokenizer.ggml.eos_token_id is [1] (<eos> only) — confirmed directly on the conversion pod. You must set <turn|> as an explicit stop string in your runtime (LM Studio: add it under the model's stop-string/sequence setting). This was verified end-to-end at generation time (see below): once given an adequate token budget, both quants stopped cleanly at <turn|> on 100%/88% of smoke-test items — the mechanism works, but only because the stop string is set explicitly. Without it, generation would run past the intended turn boundary.

Verification

Both quants were served on-pod (RTX 3090) via llama-server (no --chat-template override) and smoke-tested against 8 real held-out stimuli — the identical seed=42 validation split used for this model's own epoch-by-epoch training-time sanity generations — at both greedy and temperature 0.7/top_p 0.95, with stop=["<turn|>"] explicit in every request.

* Both Q5_K_M "unclean" cases are the same single stimulus whose natural response (5-6 fully-explained questions) exceeds even a 2048-token budget — confirmed well-formed up to the cutoff, i.e. a budget limit, not a stop-token failure.

The <turn|> stop-token risk this conversion was built around did not materialize: once given an adequate token budget, Q4_K_M reached 100% stop-clean and Q5_K_M 88% (with the sole "failure" being budget exhaustion on an unusually long response, not a metadata problem). Duplicate rates on both quants (0.013–0.171) are in the same range as this model's own HF Transformers training-time numbers (0.067–0.150), with Q4_K_M-greedy matching almost exactly (0.150) — no evidence quantization introduces or worsens repetition.

Q4_K_M does show real, reproducible structural degradation that Q5_K_M mostly avoids: on specific stimuli, Q4_K_M consistently produces a well-formed numbered question set with 4 choices each, but omits Correct:/Explanation: entirely for every question, identically at both greedy and temp 0.7 (ruling out decoding-randomness noise). Q5_K_M shows almost none of this. This was not cross-checked against the unquantized fp16 model on these same stimuli, so whether this is introduced by Q4_K_M's quantization specifically or is a pre-existing model weak point on this stimulus type is an open question — flagged rather than overclaimed. Full per-item results and raw malformed outputs in this repo's gguf_verification.md.

Decision: start local use from Q5_K_M — Q4_K_M showed malformed output on this smoke test that Q5_K_M did not.

Recommended local config

• Full GPU offload (-ngl 99 or LM Studio's "GPU offload: max") — both quants fit comfortably on a 12GB card
• Context length: -c 4096
• Chat template: leave on default / embedded (see above — do not override)
• Stop string: add <turn|> explicitly (see above — not in metadata)
• Given the structural-adherence gap above, prefer Q5_K_M for anything beyond casual testing

Task

Source excerpt (primary or secondary source) in → a set of AP US History-style multiple-choice questions out (2-8 questions per set observed in training data, each with a stem, 4 choices A-D, a Correct: key, and an Explanation:).

Training provenance

• Base: Gemma 4 (12B), continued from ambrosfitz/gemma4-ushistory-sft-parityfix
• SFT: LoRA (r=32, alpha=64) on ambrosfitz/apush-synthetic-mcq (895 synthetic train / 47 held-out val, seed=42; a separate 286-set authentic eval split exists in the dataset for answering-accuracy evaluation, not used in this generation-focused fine-tune)
• Masking: Unsloth's train_on_responses_only (marker-string based, independent of chat-template {% generation %} tags), with a blocking step-0 mask gate that verified the mask structurally correct before training started — assistant_only_loss was deliberately left unset, unlike the prior gemma4-ushistory-apush-continued run whose masking status could not be determined post-hoc
• Result: intra-set duplicate rate dropped roughly an order of magnitude versus the prior (mask-unverified) run, across all 3 epochs
• Merged epoch-3 model → this GGUF conversion

Full training report: TRAIN_REPORT.md in the source repo.