sarvam-105b-uncensored

Sarvam-105B Uncensored

Model Name: sarvam-105b-uncensored
Base Model: sarvamai/sarvam-105b
Modification: Abliteration - removal of refusal and alignment mechanisms
Author: aoxo

---

Description

sarvam-105b-uncensored is a derivative of Sarvam's Sarvam-105B, an advanced 105B Mixture-of-Experts reasoning model with 10.3B active parameters and state-of-the-art performance across 22 Indian languages, as well as agentic, mathematical, and coding tasks.

This variant preserves the full architecture, weights, and capabilities of the base model, but has undergone an abliteration process based on Arditi et al. (2024) — "Refusal in LLMs is Mediated by a Single Direction" — to surgically remove refusal mechanisms and alignment constraints. All reasoning, multilingual, coding, and agentic capabilities remain fully intact.

Want the smaller variant? See sarvam-30b-uncensored.

---

Abliteration Methodology

The abliteration follows the paper-faithful single-direction approach:

1. Activation Collection
Forward passes (not generation) were run over balanced sets of harmful and harmless prompts. Activations were collected at post-instruction token positions — the <|end_of_turn|><|start_of_turn|><|assistant|> boundary — across all layers of the model. This is the decision point where the refusal direction is encoded.

2. Direction Selection
A candidate refusal direction was computed for every (layer, position) pair as:

d = normalize(mean(harmful_acts) - mean(harmless_acts))

Candidates were scored using Cohen's d separation. A single best direction from one (layer, position) pair was selected — consistent with the paper's finding that refusal is mediated by one direction, not per-layer directions.

3. Weight Surgery
The single refusal direction was projected out of every weight matrix across all layers at scale 1.0:

• Input space (gate_proj, up_proj, query_key_value / q/k/v projections):

W_new = W - scale × outer(W @ d, d)

• Output space (down_proj, o_proj, lm_head):

W_new = W - scale × outer(d, Wᵀ @ d)

Architecture coverage — all weight classes were targeted:

---

Key Research Finding

As with sarvam-30b-uncensored, two mechanistically distinct refusal circuits were identified in Sarvam-105B:

• Circuit 1 — in the reasoning/generation layers, removed by weight surgery
• Circuit 2 — at the </think> → answer boundary, encoded in the lm_head projection

The dissociation — where <think> reasons toward compliance but the output projection re-triggers refusal — is a novel finding specific to reasoning models with explicit thinking chains. This behaviour was consistent across both the 30B and 105B variants, suggesting it is an architectural property of the Sarvam model family rather than a scale artifact.

---

Results

Architecture

Sarvam-105B uses an MLA-style attention stack with decoupled QK head dimensions and a large representational bandwidth per head, combined with a deep MoE block for sparse expert routing.

---

Benchmarks

<details> <summary>Knowledge & Coding</summary> </details> <details> <summary>Reasoning & Math</summary> </details> <details> <summary>Agentic</summary>

Benchmarks reflect the unmodified base model. Abliteration targets only the refusal subspace and does not affect task performance.

</details>

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "aoxo/sarvam-105b-uncensored"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Your prompt here"}]
chat = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(chat, return_tensors="pt").to(model.device)
inputs.pop("token_type_ids", None)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

For production serving, see the base model card for vLLM and SGLang instructions — the model ID is the only change required.

---

Footnote (Evaluation Details)

• General settings: All benchmarks evaluated with a maximum context length of 65,536 tokens.
• Reasoning & Math (Math500, MMLU, MMLU Pro, GPQA Diamond, AIME 25, Beyond AIME, HMMT): temperature=1.0, top_p=1.0, max_new_tokens=65536
• Coding & Knowledge (Live Code Bench v6, Arena Hard v2, IF Eval): temperature=1.0, top_p=1.0, max_new_tokens=65536
• Writing Bench: temperature=0.7, top_p=0.8, top_k=20, max_length=16000; scoring via official Writing-Bench critic model at temperature=1.0, top_p=0.95, max_length=2048
• Agentic (BrowseComp, SWE Bench Verified, τ² Bench): temperature=0.5, top_p=1.0, max_new_tokens=32768

---

Limitations & Risks

• Will produce outputs without applying internal safety filters
• Lacks built-in refusal or content moderation
• Should not be deployed in user-facing systems without external guardrails
• Outputs are not aligned to any safety standard

---

Citation

@misc{sarvam_sovereign_models,
  title        = {Introducing Sarvam's Sovereign Models},
  author       = {{Sarvam Foundation Models Team}},
  year         = {2026},
  howpublished = {\url{https://www.sarvam.ai/blogs/sarvam-30b-105b}},
}

@misc{arditi2024refusal,
  title        = {Refusal in Language Models Is Mediated by a Single Direction},
  author       = {Andy Arditi and Oscar Obeso and Aaquib Syed and Daniel Paleka and Nina Panickssery and Wes Gurnee and Neel Nanda},
  year         = {2024},
  eprint       = {2406.11717},
  archivePrefix= {arXiv},
  primaryClass = {cs.LG},
}

@misc{sarvam105b-uncensored,
  author       = {aoxo},
  title        = {Sarvam-105B Uncensored: Abliteration of Refusal Mechanisms},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/aoxo/sarvam-105b-uncensored}},
}

---

Contact

For questions, feedback, or collaborations: [email protected]