Downloads · 30 days
24
17% of all-time downloads
blascotobasco/Gemma-4-14B-A4B-Heretic
Gemma-4-14B-A4B-Heretic is a machine learning model from blascotobasco. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Following on my experiments with restoring pruned models, this is Gemma 4 26B A4B pruned down from 128 to 64 experts with a significantly more effective pruning strategy than the previous attempt, and then given GaLor…
Downloads · 30 days
24
17% of all-time downloads
All-time downloads
139
Public
Parameters
14.4B
52.3 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors28.8 GB · 55%
From the Hugging Face model README
Following on my experiments with restoring pruned models, this is Gemma 4 26B A4B pruned down from 128 to 64 experts with a significantly more effective pruning strategy than the previous attempt, and then given GaLore based full parameter training on only the LIMO and LIMA datasets.
The model has proven to be remarkably resilient to pruning, and while it failed in most complex tasks post prune, the training has repaired its ability to roleplay, follow instructions and logical reasoning. However a major caveat of this training run being so short is that the model struggles heavily in math. Asking a complex arithmetic question may cause gibberish or otherwise nonsensical output at worst, and the model being incorrect or giving up at best.
However the model was rigorously tested on both roleplaying and mathematics at the same time within one prompt, and while it struggles with math aside from basic calculations, it is exceptional at staying in character.
This model should be much higher quality than my previous attempt, but do not rely on it for complex tasks as a pruned model is always susceptible to errors.
The prune was based on a Heretic abliterated version of Gemma 4 26B A4B IT.