gemma-4-e4b-creative-DFT-exp

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>

goated software^^

DISCLAIMER

I am not a mathematician nor a professional coder. This was an experiment (with the help of AI of course).

This is a custom-trained version of Google's Gemma 4 E4B intended for creative writing. It was trained using a custom implementation of Distribution Fine-Tuning (DFT) designed to mathematically penalize and eliminate repetitive AI slop and predictable phrasing.

The core training algorithm was inspired by the concepts outlined in the May 18, 2026 blog post, Fixing LLM Writing with Distribution Fine-Tuning.

Standard Supervised Fine-Tuning (SFT) and RLHF often cause models to regress to a generic, hyper-structured average of human text. To counter this, this model was trained by injecting a macro-statistical loss penalty into the backpropagation loop. By calculating the Mean Squared Error (MSE) between the model's batch-level vocabulary distribution and a human target distribution (or a good creative dataset), the model was actively penalized for overusing AI-frequent vocabulary (e.g., "whisper", "shiver", "sheer").

The model was trained on 1.3~ Epochs and effective batch size of 96, using a mix of multiturn roleplaying and creative writing dataset.

Credits to Rosmine and Google Gemini for the idea and the implementation. Let me know what you think in the Community section!