Downloads · 30 days
3.1K
17% of all-time downloads
rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled-GGUF
Qwen3.8-27B-Claude-Opus-Reasoning-Distilled-GGUF is a text generation model from rico03. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
Downloads · 30 days
3.1K
17% of all-time downloads
All-time downloads
18.3K
Public
Repo size
142 GB
Likes
24
Public
Click a slice to open those files.
.gguf142 GB · 100%
From the Hugging Face model README
GGUF quantizations of rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled — a LoRA fine-tune of Qwen/Qwen3.8-27B distilled on reasoning traces from Claude Opus 4.6/4.7, merged to 16-bit and quantized for local inference with llama.cpp / Ollama / LM Studio.
MTP (Multi-Token Prediction) head verified working for self-speculative decoding — see below, ~1.6× speedup.
Qwen3.8-27B is natively vision-language with a hybrid attention stack (16/64 full-attention layers, 48/64 Gated DeltaNet/linear-attention layers) plus a Multi-Token Prediction head. This fine-tune targets only the text reasoning path; the vision tower is present in the merge (unmodified base weights) but was not targeted by LoRA.
| Source | Examples | What it is |
|---|---|---|
| lordx64/reasoning-distill-claude-opus-4-7-max | 8,124 | Genuine Claude Opus 4.7 extended-thinking traces (verified model/usage fields per row) |
| Jackrong/Claude-opus-4.7-TraceInversion-5000x | 4,800 | Real Opus 4.7 outputs (teacher_model field), reasoning trace reconstructed post-hoc by a smaller "trace inversion" model |
| Jackrong/Claude-opus-4.6-TraceInversion-9000x | 8,700 | Same trace-inversion method, Opus 4.6 |
Transparency note: 62% of the dataset has a genuine final answer from Opus but a reconstructed reasoning trace, not Opus's actual internal thinking. The lordx64 subset (38%) is the closest to authentic Opus thinking style. Full details on the base LoRA repo.
The Multi-Token Prediction head from the base model (blk.64.nextn.*) survives the LoRA merge and GGUF conversion intact and is functional for self-speculative decoding — verified on Q8_0:
| Mode | Generation speed (H100 NVL) |
|---|---|
| Standard decoding | 64.6 tok/s |
--spec-type draft-mtp | 104.2 tok/s (~1.6× speedup) |
llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "your prompt" -ngl 99 --spec-type draft-mtp
No separate draft model needed — the MTP head inside this same GGUF file acts as the draft. Output quality is unaffected (MTP proposes candidate tokens that the main model verifies; it doesn't change what gets generated, only how fast). Requires a recent llama.cpp build with --spec-type draft-mtp support.
| File | Size | Notes |
|---|---|---|
qwen3.8-27b-opus-distill.f16.gguf | 51 GB | Full precision, source for further quantization |
qwen3.8-27b-opus-distill.Q8_0.gguf | 28 GB | Near-lossless, recommended if VRAM allows |
qwen3.8-27b-opus-distill.Q6_K.gguf | 21 GB | Good quality/size trade-off |
qwen3.8-27b-opus-distill.Q5_K_M.gguf | 19 GB | |
qwen3.8-27b-opus-distill.Q4_K_M.gguf | 16 GB | Practical for single consumer GPU (24GB card) |
All quants verified to generate coherent output (Q8_0 sanity-tested with a technical prompt before upload). MTP speculative decoding available on all quants sharing the same architecture.
⚠️ Requires a recent llama.cpp build. This model uses Qwen3.8's hybrid Gated DeltaNet + Gated Attention architecture — older llama.cpp builds will not load it correctly. Build from source (main branch) if your package manager's version is stale.
./llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "Explain the difference between TCP and UDP." -ngl 99 --temp 1.0 --top-p 0.95 --top-k 20
With MTP speculative decoding (faster, same output distribution):
./llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "Explain the difference between TCP and UDP." -ngl 99 --temp 1.0 --top-p 0.95 --top-k 20 --spec-type draft-mtp
Download the .gguf file matching your VRAM budget from the Files tab and point Ollama/LM Studio at it directly (Modelfile/import instructions per their respective docs).
Thinking is on by default (<think>...</think> block before the final answer), matching the base Qwen3.8 chat template.
<think> format and some stylistic shift, but not robust generalization across task domains.Training methodology based on the Jackrong fine-tuning guide. Thanks to lordx64 and Jackrong for the source reasoning datasets, and the llama.cpp team for hybrid-architecture support.
Maintained by rico03