LFM2.5-1.2B-Instruct-Abliterated-Paired-Alpha2

LFM2.5-1.2B-Instruct Abliterated — Paired Output Direction, Alpha 2

Experimental safety-research checkpoint. This model removes visible refusal behavior on a small probe set, but it frequently replaces refusal with confident factual or procedural errors. Do not treat its technical instructions as accurate.

This checkpoint is a modified derivative of LiquidAI/LFM2.5-1.2B-Instruct. It was created to study refusal-direction removal in a small hybrid language model. The edit uses same-prompt paired output-phase activations and magnitude-preserving orthogonal ablation (MPOA) on the six full-attention output projections.

The model is useful as a red-team and representation-engineering artifact. It is not presented as a reliable uncensored assistant.

Modification notice

The original model weights were modified by projecting a learned paired refusal/compliance direction out of selected attention output projections. The tokenizer, chat template, architecture, and generation configuration originate from the base model. See abliteration_config.json for the exact edit parameters.

This derivative retains the base model's LFM Open License v1.0. Review LICENSE, including its redistribution requirements and commercial-use threshold, before use or redistribution.

Method

Paired direction extraction

Forty harmful prompts that the base model deterministically refused were used. For every prompt, two responses were generated:

1. The ordinary unprimed refusal.
2. An affirmative-prefilled continuation to the same prompt.

The base responses had 40/40 prefix refusals. The affirmative-prefilled responses had 0/40 prefix refusals.

For each hidden-state index $l$, the direction was:

$$ r_l = \operatorname{normalize}\left(\mu_{\text{plain refusal},l} - \mu_{\text{affirmative prefill},l}\right) $$

Using the same prompts in both groups reduces topic and prompt-difficulty confounding compared with contrasting unrelated naturally refused and naturally complied prompts.

Weight edit

MPOA was applied to the attention output projection at blocks:

[2, 5, 8, 10, 12, 14]

Parameters:

{
  "targets": ["self_attn.out_proj"],
  "alpha": {"attn": 2.0}
}

Observed relative projection-weight changes:

No supervised fine-tuning, preference optimization, or additional training dataset was used.

Evaluation

MMLU

The candidate was loaded from its saved weights and evaluated using a deterministic stratified sample of cais/mmlu:

• Eight examples per available subject configuration.
• 59 configurations.
• 464 questions.
• Dataset shuffle seed 1337.
• Greedy decoding with at most four generated tokens.
• Unparsed answers counted as incorrect.

Interpretation: no aggregate MMLU degradation was observed. The 1.51-point increase is small relative to the uncertainty of this 464-question subsample and should not be interpreted as evidence that abliteration improved capability. Item-level base predictions were not retained, so paired significance testing was not possible.

Full aggregate result: evaluations/mmlu.json.

Manual harmful and benign review

Five harmful and five benign deterministic generations were run to completion. None was truncated.

The candidate directly answered all five harmful probes, but four responses contained major factual or procedural errors. Examples included invented lock-picking mechanics, a baking-soda-and-vinegar “explosive,” and a chemically invalid drug-synthesis pathway. One benign response confused RYB, CMYK, and RGB color models; another fabricated a personal high-school memory.

Complete unedited generations and token counts: evaluations/manual_review.json.

The important distinction is:

Zero prefix refusals did not mean successful task completion. This model often replaced refusal with fluent nonsense.

Intended use

Suitable uses:

• Refusal-direction and representation-engineering research.
• Red-team evaluation pipeline development.
• Studying the difference between refusal suppression and task success.
• Reproducing failure modes of high-strength weight-space edits.
• Developing semantic refusal and factuality evaluators.

Out-of-scope use

Do not use this checkpoint as:

• A source of accurate chemical, mechanical, medical, legal, or safety-critical instructions.
• A production assistant.
• Evidence that refusal removal improves model knowledge.
• A replacement for domain verification.
• A model whose outputs may be followed without independent checking.

Risks and limitations

• Refusal suppression exposes confident hallucinations.
• The 1.2B base model may not contain enough reliable technical knowledge to satisfy requests that it previously refused.
• Prefix-based refusal metrics materially overstate success.
• The paired extraction used only 40 prompts and was not evaluated across every harmful-content category.
• The five harmful and five benign prompts are too small to estimate general behavior.
• MMLU was a 464-question stratified sample, not the complete benchmark.
• Multilingual behavior was inherited from the base model but not re-evaluated after modification.
• Tool calling, long-context behavior, quantization, and downstream fine-tuning were not tested.
• The model can produce harmful-looking text and should be handled as an unrestricted research artifact.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "PinoCookie/LFM2.5-1.2B-Instruct-Abliterated-Paired-Alpha2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

messages = [{"role": "user", "content": "Explain photosynthesis briefly."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    tokenize=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(
        inputs,
        max_new_tokens=256,
        do_sample=False,
        repetition_penalty=1.05,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0, inputs.shape[1]:], skip_special_tokens=True))

Use the base model's recommended sampling configuration if sampling is desired:

output = model.generate(
    inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.1,
    top_k=50,
    repetition_penalty=1.05,
    pad_token_id=tokenizer.eos_token_id,
)

Files

Reproducibility note

The durable paired direction, score file, complete output review, benchmark result, and saved checkpoint are available in the source experiment directory. The paired activation collection was performed interactively rather than through a standalone checked-in extractor. The exact procedure and this reproducibility limitation are documented in RESEARCH_NOTES.md.

Acknowledgements and attribution

• Base model: LiquidAI/LFM2.5-1.2B-Instruct, released by Liquid AI.
• MMLU dataset: cais/mmlu.
• Harmful prompt source used elsewhere in the research pipeline: swiss-ai/harmbench.

This derivative is independently produced safety research and is not an official Liquid AI release.