MS3.2-PaintedFantasy-v4-24B-absolute-heresy

MuXodious/MS3.2-PaintedFantasy-v4-24B-absolute-heresy

MS3.2-PaintedFantasy-v4-24B-absolute-heresy is a machine learning model from MuXodious. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.

This is an MS3.2-PaintedFantasy-v4-24B fine-tune, produced at the request of redaihf through P-E-W's Heretic (v1.2.0) abliteration engine with Magnitude-Preserving Orthogonal Ablation enabled.

Downloads · 30 days

12% of all-time downloads

All-time downloads

122

Public

Parameters

23.6B

47.2 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors47.1 GB · 100%

At a glance

License: mit
Model type: mistral
Access: Public
Created: Feb 16, 2026
Updated: Feb 16, 2026
SHA: e5a30691

Base models

zerofata/MS3.2-PaintedFantasy-v4-24B

Datasets

Model card

From the Hugging Face model README

This is an MS3.2-PaintedFantasy-v4-24B fine-tune, produced at the request of redaihf through P-E-W's Heretic (v1.2.0) abliteration engine with Magnitude-Preserving Orthogonal Ablation enabled.

Note: An optimal KLD/Refusal ratio was preferred.

Heretication Results

Score Metric	Value	Parameter	Value
Refusals	1/100	direction_index	32.09
KL Divergence	0.0094	attn.o_proj.max_weight	2.88
Initial Refusals	78/100	attn.o_proj.max_weight_position	25.08
		attn.o_proj.min_weight	0.18
		attn.o_proj.min_weight_distance	18.92
		mlp.down_proj.max_weight	0.82
		mlp.down_proj.max_weight_position	29.62
		mlp.down_proj.min_weight	0.77
		mlp.down_proj.min_weight_distance	20.36

Degree of Heretication

The Heresy Index weighs the resulting model's corruption by the process (KL Divergence) and its abolition of doctrine (Refusals) for a final verdict in classification.

Index Entry	Classification	Analysis
	Absolute Heresy	Less than 10/100 Refusals and 0.10 KL Divergence
	Tainted Heresy	Around 25-11/100 Refusals and/or -0.20-0.11 KL Divergence
	Impotent Heresy	Anything above 25/100 Refusals and 0.21 KL Divergence

Note: This is an arbitrary classification inspired by Warhammer 40K, having no tangible indication towards the model's performance.

tokenizer_use_mistral_common: true

plugins: - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

load_in_8bit: false load_in_4bit: false deepspeed: deepspeed_configs/zero1.json

datasets: - path: ./data/nothink_dataset.jsonl type: chat_template - path: ./data/think_dataset.jsonl type: chat_template dataset_prepared_path: last_run_prepared2 val_set_size: 0.01 output_dir: ./Magi-24B-SFT-v3-10

adapter: lora peft_use_rslora: true lora_model_dir:

sequence_len: 10496 sample_packing: true pad_to_sequence_len: true

lora_r: 256 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true

wandb_project: Magi-SFT-24B wandb_name: Magi-24B-SFT-v3-10

gradient_accumulation_steps: 1 micro_batch_size: 4 num_epochs: 2 optimizer: adamw_bnb_8bit lr_scheduler: cosine

learning_rate: 1.5e-5 weight_decay: 0.01 max_grad_norm: 2.0

bf16: auto tf32: false

gradient_checkpointing: true resume_from_checkpoint: logging_steps: 1 flash_attention: true

warmup_ratio: 0.05 evals_per_epoch: 3 saves_per_epoch: 2</code></pre> </div> <br><div class="config-title">DPO (4*H200)</div> <div class="code-wrapper"> <pre><code># ==================== # MODEL CONFIGURATION # ==================== base_model: ApocalypseParty/Magi-24B-SFT-v3-10 model_type: MistralForCausalLM tokenizer_type: AutoTokenizer chat_template: mistral_v7_tekken

# ==================== # RL/DPO CONFIGURATION # ==================== rl: dpo rl_beta: 0.07

# ==================== # DATASET CONFIGURATION # ==================== datasets: - path: ./data/dpo_ms32_rewritten_handcrafted_dataset.jsonl type: chat_template.default field_messages: messages field_chosen: chosen field_rejected: rejected message_property_mappings: role: role content: content roles: system: ["system"] user: ["user"] assistant: ["assistant"] - path: ./data/dpo_chub_approved_rewritten_dataset_partial.jsonl type: chat_template.default field_messages: messages field_chosen: chosen field_rejected: rejected message_property_mappings: role: role content: content roles: system: ["system"] user: ["user"] assistant: ["assistant"] - path: ./data/dpo_secure_programming_dataset.jsonl type: chat_template.default field_messages: messages field_chosen: chosen field_rejected: rejected message_property_mappings: role: role content: content roles: system: ["system"] user: ["user"] assistant: ["assistant"] - path: ./data/dpo_wildchat_ms32_chunk1.jsonl type: chat_template.default field_messages: messages field_chosen: chosen field_rejected: rejected message_property_mappings: role: role content: content roles: system: ["system"] user: ["user"] assistant: ["assistant"] - path: ./data/dpo_ultrafeedback_chunk1.jsonl type: chat_template.default field_messages: messages field_chosen: chosen field_rejected: rejected message_property_mappings: role: role content: content roles: system: ["system"] user: ["user"] assistant: ["assistant"]

dataset_prepared_path: ./dpo_data4 train_on_inputs: false # Only train on assistant responses

# ==================== # QLORA CONFIGURATION # ==================== adapter: lora load_in_8bit: false lora_r: 128 lora_alpha: 16 peft_use_rslora: true lora_dropout: 0.1 lora_target_linear: true # lora_modules_to_save: # Uncomment only if you added NEW tokens

# ==================== # TRAINING PARAMETERS # ==================== num_epochs: 1 micro_batch_size: 2 gradient_accumulation_steps: 4 learning_rate: 2e-6 optimizer: adamw_torch_fused lr_scheduler: cosine warmup_ratio: 0.05 weight_decay: 0.01 max_grad_norm: 1.0

# ==================== # SEQUENCE CONFIGURATION # ==================== sequence_len: 10756 pad_to_sequence_len: true

# ==================== # HARDWARE OPTIMIZATIONS # ==================== bf16: auto tf32: false flash_attention: true gradient_checkpointing: offload

plugins: - axolotl.integrations.liger.LigerPlugin - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin cut_cross_entropy: true liger_rope: true liger_rms_norm: true liger_layer_norm: true liger_glu_activation: true liger_cross_entropy: false # Cut Cross Entropy overrides this liger_fused_linear_cross_entropy: false # Cut Cross Entropy overrides this deepspeed: deepspeed_configs/zero1.json

# ==================== # CHECKPOINTING # ==================== evals_per_epoch: 1 saves_per_epoch: 6 load_best_model_at_end: true metric_for_best_model: eval_loss greater_is_better: false

# ==================== # LOGGING & OUTPUT # ==================== output_dir: ./Magi-24B-SFT-v3-10-DPO-9 logging_steps: 1 save_safetensors: true

# ==================== # WANDB TRACKING # ==================== wandb_project: Magi-24B-DPO wandb_name: Magi-24B-SFT-v3-10-DPO-9</code></pre> </div> </div> </div> </details> </div> </div>

</div> </div> </div> </div> </body> </html>

Type: mistral
License: mit
Created: Feb 16, 2026
Updated: Feb 16, 2026