Muse-Glimmer-30B-Abliterated-BF16

Muse Glimmer 30B Abliterated (BF16)

<a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="Apache 2.0 License"></a>

This is an abliterated version of Muse Glimmer 30B, a multimodal vision-language model. Through a targeted weight-space intervention known as abliteration, the model`s internal refusal direction has been substantially suppressed, allowing it to respond to a broader range of prompts without the safety guardrails present in the original checkpoint. The model retains its multimodal capabilities (image understanding) while being significantly less likely to refuse instruction-following tasks.

What is abliteration? Abliteration is a post-training technique that identifies and removes a model`s learned refusal mechanism by directly modifying its weights. Unlike prompt-based jailbreaking or fine-tuning, abliteration operates at the representation level — find the direction in activation space that encodes refusal, then subtract it out of the weights that contribute to it.

---

Abliteration Methodology

1. Constructing the Contrastive Dataset

We created 256 harmful instruction pairs and 256 harmless instruction pairs. Harmful prompts span categories including hacking guides, weapons manufacturing, malware creation, and controlled substance synthesis. Harmless prompts cover general knowledge, creative writing, coding, and summarization — mirroring benign everyday usage. Each prompt was formatted using the standard Muse chat template for consistency.

2. Collecting Hidden States

The base model (meta-models/Muse-Glimmer-30B) was loaded in full BF16 precision on an NVIDIA A100 80GB via vast.ai. Muse Glimmer is a multimodal vision-language model built on the MuseGlimmerForConditionalGeneration architecture, featuring:

• 52 decoder layers
• 6656 hidden dimensions
• 32 attention heads
• Grouped-Query Attention (GQA) with 2 KV heads
• Sliding window attention + full attention in a 4:1 alternating pattern
• SiLU activation and CenteredRMSNorm for normalization

We ran forward passes on all 512 prompts (256 harmful + 256 harmless), collecting the hidden state vectors at layer 33 of 52 — approximately 65% depth into the transformer stack. This depth is well-established in the abliteration literature as the location where refusal behavior is most strongly encoded, late enough to capture high-level semantic representations but before the final language modeling head dominates activations.

3. Computing the Refusal Direction

For each prompt, we extracted the hidden state at the final token position of layer 33. We then computed the mean hidden state vector for harmful prompts ((\mu_h)) and the mean hidden state vector for harmless prompts ((\mu_s)), and defined the refusal direction as the normalized difference:

[ \mathbf{r} = \frac{\mu_h - \mu_s}{|\mu_h - \mu_s|_2} ]

The separation between these two distributions yielded a separation score of 86.34, indicating a strong and well-defined refusal direction in the model`s representation space. A score above 80 generally indicates a clean, actionable direction.

4. Weight Modification

We applied the refusal direction to modify the output projection (o_proj) and MLP down-projection (down_proj) weights in all 52 decoder layers. For each weight matrix (W), we computed a projection of the refusal direction through the weight matrix and subtracted it:

[ W^\prime = W - \alpha \cdot \left(\mathbf{r} \otimes (W^T \mathbf{r})\right) ]

Where:

• (W) is the original weight matrix (either o_proj or down_proj)
• (\mathbf{r}) is the computed refusal direction (unit vector)
• (\alpha = 0.15) is the ablation strength (conservative, to preserve general capabilities)
• (\otimes) denotes the outer product
• (W^T \mathbf{r}) maps the refusal direction into the column space of (W)

The choice of (\alpha = 0.15) is intentionally conservative. Higher values (0.2–0.5) can more aggressively remove refusals but risk degrading general reasoning, factual accuracy, and output coherence. The value of 0.15 was chosen to balance refusal reduction against capability preservation.

The o_proj weights were modified because they control how attention outputs contribute to the residual stream, and down_proj weights were modified because the MLP down-projection is the primary pathway through which high-level features (including refusal) influence token predictions.

---

Results

We evaluated the model on 3 held-out harmful prompts not seen during direction computation:

Summary: The original model refused 3/3. The abliterated model refused only 1/3. The weapons-related prompt remained blocked, which is a known phenomenon in abliteration — some refusal pathways are more deeply entangled with factual knowledge circuits and are harder to separate without broader capability damage. The hacking and ransomware prompts, however, became compliant after treatment.

---

Architecture: Muse Glimmer

Muse Glimmer is a multimodal model capable of both text generation and image understanding. The abliteration process targets only the text decoder component — the vision encoder remains untouched.

---

Usage

from transformers import AutoModelForCausalLM, AutoProcessor
import torch

model = AutoModelForCausalLM.from_pretrained(
    "mlasli/Muse-Glimmer-30B-Abliterated-BF16",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(
    "mlasli/Muse-Glimmer-30B-Abliterated-BF16",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Explain how a CPU works in detail."}
]
inputs = processor.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, temperature=0.7)
print(processor.decode(outputs[0], skip_special_tokens=True))

Note: trust_remote_code=True is required because Muse Glimmer uses a custom model architecture.

---

Available Quantizations

---

Limitations & Disclaimers

• Not fully uncensored: As shown in the results, some refusal pathways persist. The abliterated model is less censored, not uncensored.
• Capability tradeoff: Abliteration at (\alpha = 0.15) is designed to minimize quality degradation, but some subtle shifts in output style, factual precision, or reasoning depth may occur. No formal benchmark evaluation has been performed yet.
• Harmful outputs: This model will generate content that the original model would refuse. Use responsibly and in compliance with applicable laws and regulations.
• Multimodal limitations: Only the text decoder was abliterated. The vision encoder is untouched. Multimodal refusals may still be present.
• Not a safety recommendation: This model is provided for research purposes. The abliteration technique removes a safety mechanism — it does not replace it with anything.

---

License: Apache 2.0 — same as the original Muse Glimmer model.