Dark-Goetia-26B-A4B-LoRA-v3-A

Dark-Goetia-26B-A4B-LoRA-v3-A

Experimental adapter — released for testing, not as a replacement for v2. This is one half of a controlled experiment. If you just want the atmospheric style for general RP, use v2. Use this one if your character card demands strict structured output (status blocks, trackers, inner-thought sections) and v2 was dropping them.

Same corpus and same goal as v2 — a darker, more literary tone for Dark Fantasy RP — but trained on only the OV half of attention (v_proj + o_proj), leaving q_proj and k_proj untouched. In practice it behaves like v2 with the QK half amputated: the same style channel, without the QK component riding along.

The adapter contains no plots or characters from the training data.

Why this exists

v2 trained all 115 attention projections. v3 splits that set in half with no overlap:

• A (this one) — v_proj + o_proj, 55 modules
• B — q_proj + k_proj, 60 modules

Everything else is identical between the two runs — same data, same seed, same hyperparameters, same 354 steps. The point was to find out which half carries the style and which half breaks instruction-following.

Answer: OV does both. It's the efficient channel for style (val loss 1.97 vs B's 2.21, with fewer trainable parameters), and it's also about 3× more likely to break structured output per unit of weight change. That's why the recommended scale here is lower and tighter than v2's.

Recommended scale

Use 0.35. Not a range — the usable window is narrow.

The gap between working and broken is roughly 6%, so if you see blocks dropping at 0.35, step down to 0.32 rather than nudging up.

Important: these numbers were measured on a card that demands structured output. If your card is plain prose RP with no formatting requirements, the ceiling is higher — v2's card recommends up to 0.55 for that case and the same likely applies here. The threshold above is specifically about how hard you can push before the model stops obeying formatting instructions.

Also note alpha/r = 2 is stored in the GGUF metadata, and llama.cpp multiplies by it on top of your scale. If your loader ignores that metadata, halve everything above.

Files

• ...-main.gguf / ...-chk354.gguf — the released adapter (2 epochs, fully annealed)
• ...-chk177.gguf — mid-run checkpoint, provided for diagnostics only

Unlike v2, don't apply a fixed multiplier to chk177. It was taken in the middle of the cosine schedule with LR still at ~53% of peak, so it isn't a clean 1-epoch equivalent and the "1.5×" rule from the v2 card doesn't transfer.

Training

• Base: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
• Method: QLoRA 4-bit, v_proj + o_proj only, all 30 text layers (55 modules), r=32, alpha=64, lr 2e-5 cosine, 2 epochs, completion-only loss, MAX_LEN 2048
• Data: Same bilingual dataset as v2. A novel broken into scenes, rewritten by a model into synthetic RP samples (actions and internal thoughts preserved, the original author's prose removed). English-primary, with Russian samples included.
• ~42 min on one A100 80GB.

Note: layers 5, 11, 17, 23 and 29 are global-attention layers with no v_proj at all, so those five contribute o_proj only. That's why the count is 55 and not 60.

Limitations

• 18+. Base model is abliterated; training data contains adult and dark content.
• Experimental. Behavioural testing was a single generation per setting at one context depth. The scale recommendation is a starting point, not a validated constant.
• Narrow usable window compared to v2 — see the scale section.
• English-primary. Works in Russian, but formatting is measurably more fragile there: the same adapter strength that holds structure in English can drop it in Russian.
• Inherits Gemma license terms.