Qwen2.5-14B-Instruct-abliterated-exl2

Qwen2.5-14B-Instruct-abliterated-exl2

Model: Qwen2.5-14B-Instruct-abliterated
Made by: huihui-ai

Quants

4bpw h6 (main)
4.5bpw h6
5bpw h6
6bpw h6
Didn't make 8bpw.

Quantization notes

I accidentally made these quants and didn't finish 8bpw after noticing v2 version, that's why 8bpw quant is missing.

Made with Exllamav2 0.2.3 with the default dataset. These require modern RTX cards on Windows/Linux or AMD on Linux.
The model have to fit the GPU to work properly. For example RTX3060/12GB should be able to load 4.5-5bpw/Q6 cache and 16k context.
It requires an app with Exllamav2 loader, such as Text-Generation-WebUI, TabbyAPI and some others.

Original model card

huihui-ai/Qwen2.5-14B-Instruct-abliterated

This is an uncensored version of Qwen/Qwen2.5-14B-Instruct created with abliteration (see this article to know more about it).

Special thanks to @FailSpy for the original code and technique. Please follow him if you're interested in abliterated models.

Important Note There's a new version available, please try using the new version Qwen2.5-14B-Instruct-abliterated-v2.

Usage

You can use this model in your applications by loading it with Hugging Face's transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "huihui-ai/Qwen2.5-14B-Instruct-abliterated"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Initialize conversation context
initial_messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}
]
messages = initial_messages.copy()  # Copy the initial conversation context

# Enter conversation loop
while True:
    # Get user input
    user_input = input("User: ").strip()  # Strip leading and trailing spaces

    # If the user types '/exit', end the conversation
    if user_input.lower() == "/exit":
        print("Exiting chat.")
        break

    # If the user types '/clean', reset the conversation context
    if user_input.lower() == "/clean":
        messages = initial_messages.copy()  # Reset conversation context
        print("Chat history cleared. Starting a new conversation.")
        continue

    # If input is empty, prompt the user and continue
    if not user_input:
        print("Input cannot be empty. Please enter something.")
        continue

    # Add user input to the conversation
    messages.append({"role": "user", "content": user_input})

    # Build the chat template
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    # Tokenize input and prepare it for the model
    model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

    # Generate a response from the model
    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=8192
    )

    # Extract model output, removing special tokens
    generated_ids = [
        output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]
    response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

    # Add the model's response to the conversation
    messages.append({"role": "assistant", "content": response})

    # Print the model's response
    print(f"Qwen: {response}")

Evaluations

Evaluation is ongoing, to be continued later.