Downloads · 30 days
52
1% of all-time downloads
waguriagent/qwen2.5-7b-uncensored-gguf
qwen2.5-7b-uncensored-gguf is a text generation model from waguriagent. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
A LoRA fine-tune of Qwen2.5-7B-Instruct, merged into the base weights and quantized to Q4KM GGUF for efficient local inference. Runs comfortably on consumer GPUs with 8 GB VRAM (e.g. RTX 4060) via Ollama or llama.cpp.
Downloads · 30 days
52
1% of all-time downloads
All-time downloads
3.5K
Public
Repo size
4.7 GB
Likes
1
Public
Click a slice to open those files.
.gguf4.7 GB · 100%
From the Hugging Face model README
A LoRA fine-tune of Qwen2.5-7B-Instruct, merged into the base weights and quantized to Q4_K_M GGUF for efficient local inference. Runs comfortably on consumer GPUs with 8 GB VRAM (e.g. RTX 4060) via Ollama or llama.cpp.
| Property | Value |
|---|---|
| Base model | Qwen/Qwen2.5-7B-Instruct |
| Architecture | Qwen2 (7.6B parameters) |
| Fine-tuning method | LoRA (rank 32, alpha 64) |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Precision (training) | BF16 |
| Quantization | Q4_K_M (4-bit, k-quant medium) |
| File size | ~4.4 GB |
| Context length | 32,768 tokens (inherited from base) |
| Prompt format | ChatML (Qwen2.5) |
This repository ships the Q4_K_M quant — the recommended balance of size and quality for most use cases. It keeps the most sensitive weights at higher precision while compressing the rest to 4-bit, yielding minimal quality loss versus the full-precision model at roughly a quarter of the size.
| Quant | Bits | Size | Quality | RAM/VRAM |
|---|---|---|---|---|
| Q4_K_M | ~4.5 | 4.4 GB | Good (recommended) | ~6 GB |
Pull and run directly from the Hub:
ollama run hf.co/waguriagent/qwen2.5-7b-uncensored-gguf:Q4_K_M
Or build a local model from the downloaded GGUF with a Modelfile:
FROM ./qwen2.5-7b-uncensored.Q4_K_M.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
ollama create my-model -f Modelfile
ollama run my-model
# Download the GGUF
huggingface-cli download waguriagent/qwen2.5-7b-uncensored-gguf \
qwen2.5-7b-uncensored.Q4_K_M.gguf --local-dir .
# Run interactively
./llama-cli -m qwen2.5-7b-uncensored.Q4_K_M.gguf \
-p "You are a helpful assistant." -cnv
# Or serve an OpenAI-compatible API
./llama-server -m qwen2.5-7b-uncensored.Q4_K_M.gguf -c 8192
from llama_cpp import Llama
llm = Llama(
model_path="qwen2.5-7b-uncensored.Q4_K_M.gguf",
n_ctx=8192,
n_gpu_layers=-1, # offload all layers to GPU
)
out = llm.create_chat_completion(
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain gradient descent in one paragraph."},
],
temperature=0.7,
)
print(out["choices"][0]["message"]["content"])
This model uses the Qwen2.5 ChatML template:
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
The adapter was trained with PEFT + TRL (no quantization during training), loading the base model in BF16 and training LoRA adapters directly on an H100 80GB GPU. Sequence packing was enabled so short samples are concatenated into dense sequences, maximizing token throughput per step.
| Hyperparameter | Value |
|---|---|
| Epochs | 1 |
| Effective batch size | 32 (4 × 8 grad accumulation) |
| Sequence length | 2048 |
| Learning rate | 2e-4 |
| LR scheduler | Cosine with 3% warmup |
| Optimizer | AdamW (fused) |
| Weight decay | 0.01 |
| Max grad norm | 0.3 |
| Attention | FlashAttention-2 |
| Gradient checkpointing | Enabled |
| LoRA rank / alpha | 32 / 64 |
| LoRA dropout | 0 |
BF16 LoRA train (1 epoch)
→ merge adapter into base weights
→ convert to GGUF F16 (llama.cpp)
→ quantize to Q4_K_M
→ upload to Hugging Face
Released under the Apache 2.0 license, consistent with the Qwen2.5-7B-Instruct base model. Review the base model license for full terms.