Downloads · 30 days
8
8% of all-time downloads
ShotokanJ/Qwen3-30B-A3B-Instruct-finetune-Atlas-Think-Cot-Test
Qwen3-30B-A3B-Instruct-finetune-Atlas-Think-Cot-Test is a machine learning model from ShotokanJ. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This repository contains a lightweight PyTorch adapter trained with the ggufForge toolkit. It is designed to improve the generation quality of the ultra-low-bit quantized GGUF model unsloth/ERNIE-4.5-21B-A3B-Thinking-…
Downloads · 30 days
8
8% of all-time downloads
All-time downloads
103
Public
Repo size
470 MB
Likes
1
Public
Click a slice to open those files.
.pt470 MB · 100%
From the Hugging Face model README
This repository contains a lightweight PyTorch adapter trained with the ggufForge toolkit. It is designed to improve the generation quality of the ultra-low-bit quantized GGUF model unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF without modifying the base model itself.
The adapter implements External Logit Correction – a small transformer that refines the logits produced by llama.cpp during inference. It is not a standalone GGUF file and cannot be used directly with pure llama.cpp. Instead, you need the ggufForge inference wrapper, which loads both the base GGUF model (via llama-cpp-python) and the adapter (in PyTorch), then combines them on the fly.
ERNIE-4.5-21B-A3B-Thinking-UD-Q2_K_XL.ggufThis adapter is intended for research and experimentation with ultra-low‑bit quantized LLMs. It demonstrates that a tiny external network can recover much of the quality lost during extreme quantization, especially on MoE‑based models like ERNIE.
Do not expect this adapter to work with other base models (e.g., Llama, Mistral) – it was trained specifically for ERNIE‑4.5‑21B with the exact tokenizer and vocabulary of that GGUF file.
You need the ggufForge Python package to run inference with this adapter. The library handles downloading the base GGUF model (if not already cached), loading the adapter weights, and performing the combined forward pass.
git clone https://github.com/ShotokanOSS/ggufForge.git
cd ggufForge
python -m venv .venv
source .venv/bin/activate # or .venv\Scripts\activate on Windows
# Basic install
pip install -U pip
pip install .
# For GPU acceleration (recommended)
CMAKE_ARGS="-DLLAMA_CUDA=on" pip install .
The adapter is automatically downloaded from this Hugging Face repo when you specify --adapter-repo ShotokanJ/ERNIE-4.5-21B-A3B-Thinking-GGUF-finetune-Atlas-Think-Cot.
run-inference \
--mode single \
--question "Explain the concept of quantum entanglement in simple terms." \
--adapter-repo ShotokanJ/ERNIE-4.5-21B-A3B-Thinking-GGUF-finetune-Atlas-Think-Cot \
--base-repo unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF \
--gguf-filename ERNIE-4.5-21B-A3B-Thinking-UD-Q2_K_XL.gguf
run-inference \
--mode chat \
--question "Explain the concept of quantum entanglement in simple terms." \
--adapter-repo ShotokanJ/ERNIE-4.5-21B-A3B-Thinking-GGUF-finetune-Atlas-Think-Cot \
--base-repo unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF \
--gguf-filename ERNIE-4.5-21B-A3B-Thinking-UD-Q2_K_XL.gguf
run-inference \
--mode compare \
--question "Explain the concept of quantum entanglement in simple terms." \
--adapter-repo ShotokanJ/ERNIE-4.5-21B-A3B-Thinking-GGUF-finetune-Atlas-Think-Cot \
--base-repo unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF \
--gguf-filename ERNIE-4.5-21B-A3B-Thinking-UD-Q2_K_XL.gguf
All generation parameters (temperature, min‑p, repetition penalty, etc.) can be adjusted as described in the ggufForge documentation.
The adapter was trained using the train-adapter command from ggufForge:
train-adapter \
--model unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF \
--filename ERNIE-4.5-21B-A3B-Thinking-UD-Q2_K_XL.gguf \
--dataset prithivMLmods/Atlas-Think-Cot-12M \
--adapter-dim 256 \
--steps 14000 \
--learning-rate 5e-5 \
--batch-size 1 \
--accumulation-steps 32 \
--output-dir ernie-adapter
| Metric | Base Model | + Adapter | Improvement |
|---|---|---|---|
| Validation Perplexity | 4.39 | 3.46 | 21.2% |
| Human‑rated coherence | – | – | Noticeably better reasoning chains |
Detailed evaluation logs can be found in the ggufForge study (to be added).
For questions or issues, please open a ticket in the ggufForge issue tracker.