Downloads · 30 days
51
6% of all-time downloads
2023mt13042/Llama-3.1-8B-uncensored-abliterated
Llama-3.1-8B-uncensored-abliterated is a text generation model from 2023mt13042. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.1.
This repository contains a modified/fine-tuned variant of meta-llama/Llama-3.1-8B-Instruct processed using direction-based representation refinement (Abliteration methodology).
Downloads · 30 days
51
6% of all-time downloads
All-time downloads
894
Public
Parameters
4.6B
22.8 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors6 GB · 100%
How the weights are stored.
U83.5B · 75%
From the Hugging Face model README
This repository contains a modified/fine-tuned variant of meta-llama/Llama-3.1-8B-Instruct processed using direction-based representation refinement (Abliteration methodology).
meta-llama/Llama-3.1-8B-Instructsafetensors (fp16)You can run this model using Hugging Face transformers on a GPU with at least 16 GB of VRAM (or ~8 GB using 4-bit/8-bit quantization).
Ensure you have the required packages installed:
pip install transformers torch accelerate bitsandbytes
Use the following Python snippet to run inference using Hugging Face transformers:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "2023mt13042/Llama-3.1-8B-uncensored-abliterated"
# Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Load Model in fp16
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto"
)
# Prepare Prompt
messages = [
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "Explain quantum computing in simple terms."}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.7
)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(response)
For GPUs with lower VRAM (~8 GB VRAM), load the model with bitsandbytes 4-bit NF4 quantization:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
model_id = "2023mt13042/Llama-3.1-8B-uncensored-abliterated"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map="auto"
)
To run this model locally with Ollama:
.gguf format using llama.cpp.Modelfile with the following configuration:FROM ./llama-3.1-8b-abliterated-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.9
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>
{{ .Response }}<|eot_id|>"""
Build and launch the model in Ollama:
ollama create llama3-abliterated -f Modelfile
ollama run llama3-abliterated