granite-3.2-8b-instruct-abliterated-exl2

granite-3.2-8b-instruct-abliterated-exl2

Original model: granite-3.2-8b-instruct-abliterated by huihui-ai
Based on: granite-3.2-8b-instruct by Granite Team, IBM

Quants

4bpw h6 (main)
4.5bpw h6
5bpw h6
6bpw h6
8bpw h8

Quantization notes

Made with Exllamav2 0.2.8. These quants require Exllamav2 0.2.7 or newer, older versions aren't supported.
You can use these quants with TabbyAPI, Text-Generation-WebUI and some other apps.
Exl2 quants require Nvidia RTX GPU on Windows or Nvidia RTX/AMD ROCm on Linux.
Models have to fully fit your GPU VRAM because native RAM offloading isn't supported.

Original model card

huihui-ai/granite-3.2-8b-instruct-abliterated

This is an uncensored version of ibm-granite/granite-3.2-8b-instruct created with abliteration (see remove-refusals-with-transformers to know more about it).
This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens.

Use with ollama

You can use huihui_ai/granite3.2-abliterated directly

ollama run huihui_ai/granite3.2-abliterated

Donation

If you like it, please click 'like' and follow us for more updates.
You can follow x.com/support_huihui to get the latest model information from huihui.ai.

Your donation helps us continue our further development and improvement, a cup of coffee can do it.

• bitcoin:

  bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge