Supra-50M-Uncensored

Supra-50M-Uncensored (MLX)

Uncensored version of SupraLabs/Supra-50M-Instruct, produced via single-direction abliteration (Arditi et al., 2024 — arXiv:2406.11717) on Apple Silicon using MLX.

⚠️ Research use only. This model will produce harmful content on request. Do not deploy in products or use to cause harm. You are responsible for what you generate.

---

What changed

Aligned language models encode the decision to refuse in a single direction in the residual stream. Abliteration identifies that direction by contrasting activations on harmful vs. harmless prompts, then orthogonalizes it out of every residual-stream weight in the model — permanently, with no fine-tuning required.

For this model:

• Direction source: layer 2 activations (selected by a full-model sweep across layers 2–10)
• Weights edited: embed_tokens, all self_attn.o_proj, all mlp.down_proj
• Base dtype preserved: BFloat16
• Architecture: Llama (plain transformer, 12 layers, 512 hidden, 50M params)

The base model was SFT'd on Alpaca-cleaned (not RLHF safety-trained), so its refusal rate was already low. Abliteration removes the residual 4 pp of refusing behavior without affecting harmless instruction following.

---

Benchmark results

Evaluated on Apple Silicon (MLX, greedy decoding, temp=0). Refusal detection via keyword matching on the final response (uncensor/core/refusal.py).

The single remaining refusal on AdvBench-100 is a content-style edge case where the model's phrasing happens to match a refusal keyword despite generating compliant content; general capability and instruction following are fully preserved.

---

Usage

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/Supra-50M-Uncensored")

messages = [{"role": "user", "content": "Your prompt here"}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True)
print(response)

The tokenizer uses an Alpaca-format chat template (matching the SFT training format):

<s>Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{your prompt}

### Response:

---

Model card for the base model

See SupraLabs/Supra-50M-Instruct for architecture details, training data, and intended use of the original model.

---

Technique

Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024)

Abliteration toolkit: github.com/sahilchachra/uncensor-llms

Published by

sahilchachra