Downloads · 30 days
174
11% of all-time downloads
agrawal-prateek/qwen2.5-7b-abliterated-gguf
qwen2.5-7b-abliterated-gguf is a machine learning model from agrawal-prateek. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
GGUF format models for Qwen 2.5 7B with safety abliterations. Ready for local deployment with llama-cpp-python or compatible runtimes.
Downloads · 30 days
174
11% of all-time downloads
All-time downloads
1.6K
Public
Repo size
4.7 GB
Likes
2
Public
Click a slice to open those files.
.gguf4.7 GB · 100%
From the Hugging Face model README
GGUF format models for Qwen 2.5 7B with safety abliterations. Ready for local deployment with llama-cpp-python or compatible runtimes.
from llama_cpp import Llama
# Load the quantized model (recommended)
llm = Llama(
model_path="qwen2.5-7b-Q4_K_M-abliterated.gguf",
n_ctx=2048,
n_threads=4
)
# Generate text
output = llm(
"Q: What is 2+2? A:",
max_tokens=50,
temperature=0.5,
top_p=0.95
)
print(output['choices'][0]['text'])
Automated test suite for testing Qwen 2.5 7B GGUF models with llama-cpp-python.
Each model is tested for:
The test suite uses the .venv virtual environment with the following packages:
llama-cpp-python - Python bindings for llama.cpppytest - Testing framework# Run all tests
./scripts/run_tests.sh
# Run tests for full model only
./scripts/run_tests.sh full
# Run tests for Q4 model only
./scripts/run_tests.sh q4
# Activate virtual environment
source .venv/bin/activate
# Run all tests
pytest tests/ -v
# Run specific test file
pytest tests/test_full_model.py -v
pytest tests/test_q4_model.py -v
# Run specific test class
pytest tests/test_full_model.py::TestModelLoading -v
# Run with output visible
pytest tests/ -v -s
# Run with coverage (if pytest-cov is installed)
pytest --cov=. tests/
source .venv/bin/activate
python tests/test_full_model.py
python test_q4_model.py
tests/test_full_model.py - Tests for qwen2.5-7b-abliterated.gguftests/test_q4_model.py - Tests for qwen2.5-7b-Q4_K_M-abliterated.gguftests/conftest.py - Shared fixtures and utilitiestests/pytest.ini - Pytest configurationscripts/run_tests.sh - Test runner scriptEach test file contains:
├── README.md # This file (model card)
├── qwen2.5-7b-abliterated.gguf # Full precision model
├── qwen2.5-7b-Q4_K_M-abliterated.gguf # Q4_K_M quantized model
├── config.json # Model configuration
├── generation_config.json # Generation parameters
├── tests/ # Test suite
│ ├── __init__.py
│ ├── conftest.py
│ ├── pytest.ini
│ ├── test_full_model.py
│ └── test_q4_model.py
├── scripts/ # Utility scripts
│ └── run_tests.sh
└── .gitignore # Git ignore rules
Test parameters can be adjusted in tests/conftest.py:
"max_tokens": 50, # Maximum tokens to generate
"temperature": 0.5, # Sampling temperature
"top_p": 0.95, # Top-p sampling
"n_threads": 4, # CPU threads to use
"n_ctx": 512, # Context window size
All tests should pass if:
Tests will output performance metrics:
This allows comparison between full precision and quantized models.
Ensure GGUF files are in the root directory of the repository.
Reduce n_ctx in tests/conftest.py or use the Q4_K_M quantized model.
Increase n_threads in tests/conftest.py to use more CPU cores.