Downloads · 30 days
46
13% of all-time downloads
OpenMOSE/RWKV-Qwen3-32B-hxa07d-L8-GGUF
RWKV-Qwen3-32B-hxa07d-L8-GGUF is a machine learning model from OpenMOSE. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
OpenMOSE/RWKV-Qwen3-32B-hxa07d-L8-GGUF is a 32B-parameter instruction-tuned hybrid model built from Alibaba’s Qwen3-32B, converted into a new RWKV hxa07D + NoPEAttention architecture.
Downloads · 30 days
46
13% of all-time downloads
All-time downloads
353
Public
Repo size
49.9 GB
Likes
1
Public
Click a slice to open those files.
.gguf49.9 GB · 100%
From the Hugging Face model README
OpenMOSE/RWKV-Qwen3-32B-hxa07d-L8-GGUF is a 32B-parameter instruction-tuned hybrid model built from Alibaba’s Qwen3-32B, converted into a new RWKV hxa07D + NoPEAttention architecture.
The core design goal is simple and aggressive:
Keep attention layers to the absolute minimum (8 layers total), and make the remaining 56/64 layers linear-time RWKV, achieving 87.5% KV-cache reduction compared to a fully-attention 64-layer stack.
This model is the continuation of an iterative research direction discussed throughout earlier development threads: pushing practical long-context + high-throughput inference by combining RWKV-style O(n) recurrence with a small number of strategically placed attention layers, while relying on distillation (RADLADS) to preserve instruction-following quality and reasoning.
This model requires a custom fork of llama.cpp with RWKV07D implementation, based on mollysophia's RWKV7 implementation.
git clone https://github.com/OpenMOSE/llama.cpp
cd llama.cpp
git checkout hxa07d
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
First, identify your GPU architecture:
gfx1100gfx942gfx908Then build with the appropriate target:
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release \
&& cmake --build build --config Release -- -j 16
Note: Replace gfx1100 with your GPU's architecture code
./build/bin/llama-cli -m YOUR_MODEL_PATH --jinja -fa 1
./build/bin/llama-cli -m YOUR_MODEL_PATH --jinja -fa 1 -ctv q8_0 -ctk q8_0
./build/bin/llama-server -m YOUR_MODE_PATH --jinja -fa 1 --port 4096 -np 1 -c 65536
OpenMOSE/RWKV-Qwen3-32B-hxa07d-L8This model is built for the same “local-first high performance” direction repeatedly emphasized in prior design discussions:
In earlier work, we explored:
This release consolidates that direction into a clean architecture statement: 8 NoPEAttention layers is enough—the rest should be fast.
The network is a 64-layer stack:
This architecture intentionally limits attention depth to reduce memory pressure and keep inference scalable.
hxa07D is based on an improved RWKV v7 lineage, with additional stability and retention improvements:
The teacher → student conversion and training pipeline is based on RADLADS, proposed by SmerkyG.
While exact staging and loss composition can vary by run, the guiding principle remains:
Despite restricting NoPEAttention to only 8 layers, the model is designed to maintain strong instruction performance through distillation and architectural alignment techniques.
| Category | Benchmark | Score | Notes |
|---|---|---|---|
| Reasoning | MMLU | 78.59% | |
| Math | GSM8K | 89.01% | |
| Long Context | PassKey / Needle | 100k | KV cache advantage should show here |
This model inherits typical biases and failure modes from large-scale web-trained instruction models. Use standard best practices:
This architecture research and implementation was made possible with computing power and technical support from Recursal AI. We sincerely thank them for enabling this work.
Distillation methodology is based on RADLADS, proposed by SmerkyG, whose ideas significantly influenced the training pipeline.
If you use this model in research or products, please cite the model repository and credit the contributors and supporting organizations.
2025 OpenMOSE