GemmaWiki-Gemma-4-26b-a4b

GemmaWiki: Gemma 4 26B-A4B LoRA Adapters for Wiki Editing

This is an independent, unofficial project. It is not affiliated with, endorsed by, or produced by Google or Google DeepMind. "Gemma" is a trademark of Google DeepMind, used here only to identify the base model this project fine-tunes.

Status: v1 is released and usable. It performs well in evaluation (see Results), with one main rough edge: on very well-known book series, generated pages can drift toward the base model's own pretraining knowledge instead of sticking to the supplied chapter (see Known limitations). Treat it as a solid pilot rather than a finished product. A broader corpus and stronger chapter-grounding are planned next.

LoRA adapters (r=32, attention-only: q_proj/k_proj/v_proj/o_proj) fine-tuning Gemma 4 26B-A4B, a mixture-of-experts model, as an automated fan-wiki editor. Given a book chapter and, when one exists, an entity's current wiki page, the model produces one of three outputs: a new page, an updated page, or a classification that no change is needed, using only information available up to that chapter. Two independently trained variants are provided here: one starting from the base pretrained model, one starting from Google's instruction-tuned checkpoint. Both ship as LoRA adapters and as ready-to-run GGUF quantizations.

Structure

Every checkpoint is a standard PEFT adapter directory (adapter_config.json plus adapter_model.safetensors).

Base-backbone adapters, adapters- base/, from google/gemma-4-26b-a4b

• run1-checkpoint-{25,50,75}: initial run, trained on a single franchise from the corpus only, an unintentional data-ordering artifact. Kept for the record, not the recommended checkpoint.
• run2-checkpoint-{025..300}, run2-checkpoint-304, run2-final: the recommended series. Warm-started from run1-checkpoint-75, retrained on a corpus-proportional balance across six popular fantasy and young-adult book series, covering page creation, page update, and chapter-triage classification. run2-final (identical weights to run2-checkpoint-304) is the finished model, trained for one epoch over the full balanced dataset.

Instruction-tuned-backbone adapters, adapters- it/, from google/gemma-4-26b-a4b-it

• run3-checkpoint-{025..300}, run3-final: the same training recipe as run2, applied to Google's instruction-tuned checkpoint instead of the plain pretrained one. run3-final is the finished model. Because this backbone already understands conversational formatting, it can be prompted two ways: raw (plain text, matching the training format directly, the recommended default) and chat (through the model's own chat template). See Results and How to Use below for the tradeoffs between them.

GGUF quantizations

Ready-to-run GGUFs for both variants (merged into their respective base models in bf16, then converted with llama.cpp). Root-level files, so they work directly with llama.cpp, LM Studio, and ollama run hf.co/Calplus/GemmaWiki-Gemma-4-26b-a4b:Q4_K_M.

Decoding note: use sampling with a repetition penalty (llama.cpp defaults are fine). Pure greedy decoding can fall into repetition loops on long generations, a base-model (Gemma 4 26B-A4B) behavior that fine-tuning does not remove.

Results

All figures below come from an automated evaluation suite comparing generated text against data withheld from training. No LLM judge is involved in any of these numbers: every metric is an exact-match, classification, or statistical computation. Three request types are covered: page update, page creation, and chapter-triage classification. All of this evaluates the model within its training distribution (one franchise was held out entirely); a qualitative caveat about very well-known series is noted under Known limitations.

Component metrics:

• D score: converts held-out cross-entropy loss into a 0 to 100 scale (100 * e^(-loss)), representing how much probability the model assigns to the actual correct continuation on wiki-editing examples it never trained on.
• I stability: 100 minus the rate of degenerate outputs across a self-feeding rollout, where the model repeatedly edits its own prior output as new chapters arrive, matching how it would be used in production.
• Format: percentage of outputs with correct structure (front matter and headings for page edits, valid JSON for classification).
• Degeneration: percentage of outputs that collapsed into repetition loops or character floods. Lower is better.
• Macro-F1: macro-averaged F1 for the three-way classification task (new page, update, no change).

Composite combines these into one score: 50% D score, 25% I stability, 15% macro-F1, and 10% split between format and inverted degeneration rate. The weighting favors the properties that most directly show the model has learned the task and holds up under realistic, repeated use, with classification accuracy and basic output hygiene as secondary factors.

¹ D score for this configuration was not measured (evaluation compute went to the configurations above first); composite renormalized over the remaining three weighted terms.

Both fine-tuned variants show a large, consistent gain over their untrained starting point on every axis that measures task competence. The untrained instruction-tuned checkpoint in particular is unstable under repeated use (70.3 I stability, 56.2% degeneration in raw prompting), and fine-tuning resolves both.

General knowledge (MMLU, 342 held-out multiple-choice questions) is not part of the composite above since it measures a different property than wiki-editing competence. For reference: base 73.7%, run1 74.9%, run2 75.1%, run2 Q4_K_M 74.9% (quantization costs 0.2 points), untrained instruction-tuned backbone 46.5%, run3 39.8%. The lower figures for the instruction-tuned-backbone variants reflect the raw, non-chat-formatted prompting style this test uses rather than knowledge loss specifically.

Deployment throughput (Q4_K_M, single GPU): run2 60.2 tok/s with full GPU offload, 11.9 tok/s with partial offload; run3 67.7 and 9.5 tok/s respectively.

Known limitations

• Page content can drift toward the base model's own knowledge of very famous series. In held-out evaluation the model writes from the chapter it is given, and hand-review found no spoiler leakage. In field testing on one of the most widely known franchises in the corpus, some pages (mostly newly created ones) mixed in details Gemma already knew from its pretraining rather than sticking to the chapter. Updates to an existing page stayed closer to the source. Fine-tuning dampens this familiarity but does not erase it, so review new pages for well-known series before publishing. Stronger chapter-grounding is a goal of the next training round.
• Input-shape specialization. Both fine-tuned variants are specialized to two request shapes: chapter plus prior page, and chapter plus classification question. A request outside either shape typically produces no output rather than an attempted answer, at any prompt length. This is a direct consequence of narrow, single-purpose LoRA training. Calling code should decide which request shape applies before invoking the model; see How to Use.
• General knowledge is out of scope for the instruction-tuned-backbone variant when prompted without a chat template. See Results for the numbers and context; this does not indicate an issue with the wiki-editing task itself.
• run1 is kept for reference only. Its training data was accidentally skewed toward a single franchise; use run2 or run3 instead.
• run3's Q4_K_M quantization has one unmeasured configuration (chat-prompt D score, see footnote in Results). The configurations that were measured already answer the question of whether quantization holds up.
• This suite measures task competence, learned confidence, and mechanical output quality. It does not include a separate qualitative audit of factual grounding or spoiler avoidance in generated prose. As with any generative model, review output before publishing it to a live wiki.

How to use

Raw vs. chat mode (run3 only)

run2 has one calling convention: plain text, matching its training format. run3 supports two: raw, plain text identical to run2's convention and the recommended default, and chat, through the model's chat template. Raw mode scores modestly higher on learned confidence and much higher on output-format consistency (see Results). Chat mode is a reasonable alternative if the calling code already expects and strips conversational wrapping around the structured output.

Request shapes

The model supports exactly two request types:

• Page update or creation: a book chapter, plus the entity's existing wiki page if one exists. The model returns the new or updated page.
• Classification: a book chapter and a question asking whether given entities' pages need a new page, an update, or no change. The model returns a JSON object mapping entity names to one of the three labels.

Production integration

Because the model is specialized to exactly these two shapes, treat it as a narrow component inside a larger pipeline rather than a general-purpose agent:

• Decide, in surrounding code, which of the two request shapes applies before calling the model. Do not ask the model to infer this itself.
• Validate output before publishing it: confirm front matter and headings for page edits, or that the response parses as JSON for classification.
• Treat empty output as a signal that the request did not match a recognized shape, not as a successful no-op.

Loading the adapters (bf16, transformers plus PEFT)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-26b-a4b", dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-26b-a4b")
model = PeftModel.from_pretrained(
    base, "Calplus/GemmaWiki-Gemma-4-26b-a4b", subfolder="adapters- base/run2-final")

For run3, load the instruction-tuned base and tokenizer instead:

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-26b-a4b-it", dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-26b-a4b-it")
model = PeftModel.from_pretrained(
    base, "Calplus/GemmaWiki-Gemma-4-26b-a4b", subfolder="adapters- it/run3-final")

Running the GGUF quantizations

ollama run hf.co/Calplus/GemmaWiki-Gemma-4-26b-a4b:Q4_K_M

Or load any of the files listed above directly with llama.cpp or LM Studio.

Future work

• Broaden the training corpus beyond the current six franchises.
• Strengthen chapter-grounding so pages rely less on the base model's prior knowledge of famous series.
• Revisit the instruction-tuned variant's chat-mode formatting, which lags raw mode in the current results.

Training details

Base architecture: Gemma 4 26B-A4B, a mixture-of-experts transformer. LoRA rank 32, attention projections only (q_proj, k_proj, v_proj, o_proj). Hardware: bf16 LoRA training with FSDP across 4 A100 GPUs.

Training data: sourced from 2,287 book chapters across 7 franchises, scraped into 53,963 wiki pages and cleaned down to 32,234 articles relevant to the source books. Six of the seven franchises were combined into the corpus-proportional training mix described above; the seventh was held out entirely to test generalization to a series the model never trained on. The model was trained on a corpus of 245,509 examples, automatically audited beforehand to catch spoiler leaks in the underlying wiki-page targets.

License

Apache 2.0, inherited for the adapters and the GGUF conversions in this repository. The base model itself remains subject to Google's own Gemma terms of use.