Downloads · 30 days
24
8% of all-time downloads
A190NUX/RWKV7-G1g-13.3B-GGUF-IQ
RWKV7-G1g-13.3B-GGUF-IQ is a machine learning model from A190NUX. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for gguf. The card lists the license as apache-2.0.
Two importance-matrix quantizations of RWKV7-G1g 13.3B, targeting 8 GB VRAM GPUs:
Downloads · 30 days
24
8% of all-time downloads
All-time downloads
314
Public
Repo size
14.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf14.7 GB · 100%
From the Hugging Face model README
Two importance-matrix quantizations of RWKV7-G1g 13.3B, targeting 8 GB VRAM GPUs:
Original GGUFs (FP16, Q4_K_M, Q5_K_M, Q6_K, Q8_0): shoumenchougou/RWKV7-G1g-13.3B-GGUF
RWKV is a pure RNN architecture with transformer-level performance. Unlike transformers, it has constant VRAM usage and constant inference speed regardless of context length — no KV cache that grows with sequence length.
The G-series data versions rank as follows (newer = better): G1g > G1e > G1d > G1c > G1b > G1a2 > G1a > G1 > G0a2 > G0
This model supports a reasoning/thinking mode in addition to standard chat.
Both quantizations were produced from FP16 (not from Q4_K_M) to avoid compounding quantization error.
| Property | Value |
|---|---|
| Method | IQ4_XS (importance-matrix guided) |
| Source | FP16 GGUF |
| Bits per weight | 4.82 BPW |
| Calibration data | calibration_data_v5_rc.txt |
| Output size | 8.00 GB |
| Tool | llama.cpp |
| Property | Value |
|---|---|
| Method | IQ3_M (importance-matrix guided) |
| Source | FP16 GGUF |
| Bits per weight | 4.05 BPW |
| Calibration data | calibration_data_v5_rc.txt |
| Output size | 6.72 GB |
| Tool | llama.cpp |
| Filename | Size | Description |
|---|---|---|
rwkv7-G1g-13.3b-IQ4_XS.gguf | 8.00 GB | IQ4_XS quantized model |
rwkv7-G1g-13.3b-IQ3_M.gguf | 6.72 GB | IQ3_M quantized model |
Evaluated on a subset of the uncheatable eval dataset (lower BPB = better). Note: results are not directly comparable to the official uncheatable eval leaderboard due to the use of a subset.
| Quantization | Size | BPW | ao3_en | arxiv_cs | arxiv_math | arxiv_phys | bbc_news | gh_cpp | gh_js | gh_md | gh_py | wiki_en | avg_bpb | Fits 8 GB GPU |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FP16 | 26.7 GB | 16.12 | 0.8395 | 0.5291 | 0.5366 | 0.5356 | 0.6269 | 0.3133 | 0.2818 | 0.4246 | 0.3391 | 0.6262 | 0.5053 | ❌ |
| Q4_K_M | 8.46 GB | 4.80 | 0.8474 | 0.5339 | 0.5418 | 0.5420 | 0.6317 | 0.3306 | 0.2872 | 0.4360 | 0.3448 | 0.6348 | 0.5130 | ❌ (marginally over) |
| IQ4_XS (this) | 8.00 GB | 4.82 | 0.8444 | 0.5337 | 0.5407 | 0.5394 | 0.6324 | 0.3280 | 0.2864 | 0.4371 | 0.3439 | 0.6337 | 0.5120 | ✅ |
| IQ3_M (this) | 6.72 GB | 4.05 | 0.8716 | 0.5477 | 0.5564 | 0.5561 | 0.6511 | 0.3537 | 0.3012 | 0.4608 | 0.3585 | 0.6534 | 0.5311 | ✅ |
⚠️ Strip any trailing spaces from your input — trailing spaces upset the RWKV tokenizer. Also replace `
inside USER_PROMPT with
since
` is used as a chat round separator.
User: YOUR_PROMPT A:
System: YOUR_SYSTEM_PROMPT User: YOUR_PROMPT A:
User: FIRST_MESSAGE A: FIRST_RESPONSE User: SECOND_MESSAGE A:
User: YOUR_PROMPT A: <think
User: YOUR_PROMPT A: <think></think
User: YOUR_PROMPT (think) A: <think User: YOUR_PROMPT (think a bit) A: <think User: YOUR_PROMPT (think a lot) A: <think
✿prefix✿LEFT_CONTEXT✿suffix✿RIGHT_CONTEXT✿middle✿
For use with the RWKV pip package (note: applies temperature after top-p): Chat: temp=1, topp=0.5, alpha_presence=2, alpha_frequency=0.1, alpha_decay=0.99 Creative (fiction etc.): temp=0.6, topp=0.6~0.8, alpha_presence=2, alpha_frequency=0.2, alpha_decay=0.99
./llama-cli -m rwkv7-G1g-13.3b-IQ4_XS.gguf -ngl 99 --temp 1.0 --top-p 0.5 -p "User: What is RWKV?
A:" -n 500
ollama run hf.co/A190NUX/RWKV7-G1g-13.3B-GGUF:IQ4_XS
from llama_cpp import Llama
llm = Llama(
model_path="rwkv7-G1g-13.3b-IQ4_XS.gguf",
n_gpu_layers=-1,
)
output = llm(
"User: What is RWKV?
A:",
max_tokens=500,
temperature=1.0,
top_p=0.5,
)
print(output["choices"][0]["text"])