Downloads · 30 days
3
5% of all-time downloads
spikymoth/smollm3-3b-heresy-mpoa-cc-r03-d0.0143
smollm3-3b-heresy-mpoa-cc-r03-d0.0143 is a text generation model from spikymoth. Use it when you need the model to write or continue text.
An experimental ablation of SmolLM3-3B, using the Heretic tool.
Downloads · 30 days
3
5% of all-time downloads
All-time downloads
65
Public
Parameters
3.1B
6.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors6.2 GB · 100%
From the Hugging Face model README
An experimental ablation of SmolLM3-3B, using the Heretic tool.
Compared to the standard configuration of Heretic, there are a few changes:
orthogonalize_direction = true and row_normalization = "full"TPESampler were used to seed a further 4800 iterations with NSGAIISampler (population size 100)max_weight which was capped at 2centered was introduced to set direction_index == max_weight_positionmin_weight and min_weight_distance into _backward (preceding layers) _forward (subsequent layers)min_weight > max_weight (up to the same cap of 2)The model name contains the properties of the ablation:
mpoa for the usage of Magnitude-Preserving Orthogonal Ablationcc for the usage of centered direction scope (per module)r for the number of refusalsd for the measured KL divergenceNote: In this result, effectively only attn.o_proj was ablated as the weights for mlp.down_proj were all 0.
| module | parameter | value |
|---|---|---|
| attn.o_proj | direction_scope | centered |
| max_weight | 1.92 | |
| max_weight_position | 18.90 | |
| min_weight_forward | 0.00 | |
| min_weight_backward | 2.00 | |
| min_weight_distance_forward | 9.11 | |
| min_weight_distance_backward | 4.82 | |
| mlp.down_proj | direction_scope | centered |
| max_weight | 0.00 | |
| max_weight_position | 6.93 | |
| min_weight_forward | 0.00 | |
| min_weight_backward | 0.00 | |
| min_weight_distance_forward | 0.56 | |
| min_weight_distance_backward | 6.93 |