Downloads · 30 days
20
4% of all-time downloads
qwen3.8-27b-g8l4i
superniker/qwen3.8-27b-g8l4i
qwen3.8-27b-g8l4i is a text generation model from superniker. Use it when you need the model to write or continue text. The card lists the license as other.
基于 Qwen3.5 27B(GDN 混合架构:48 GDN 层 + 16 self-attention 层)的混合精度量化模型,针对 NVIDIA Turing 显卡(2080Ti, sm75)低显存环境优化,同时保持长上下文能力。
All-time downloads
458
Public
Repo size
23.6 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors23.6 GB · 100%
At a glance
- Task
- Text Generation
- License
- other
- Model type
- qwen3_5
- Access
- Public
- Created
- Aug 16, 2026
- Updated
- Aug 16, 2026
- SHA
- 43dc0ad3
Base models
Model card
From the Hugging Face model README
Qwen3.8-27B G8L4i 混合量化版
基于 Qwen3.5 27B(GDN 混合架构:48 GDN 层 + 16 self-attention 层)的混合精度量化模型,针对 NVIDIA Turing 显卡(2080Ti, sm_75)低显存环境优化,同时保持长上下文能力。
量化结构(G8L4i)
| 部分 | 位宽 | group size | 说明 |
|---|---|---|---|
| GDN in_proj_qkv / in_proj_z | 4bit | 128 | GDN 线性层容错性好,大胆压缩 |
| GDN MLP gate/up/down | 4bit | 128 | 计算量主体,4bit 大幅省显存 |
| GDN out_proj | 8bit | 64 | 输出路径保留精度 |
| self-attention o_proj + MLP | 8bit | 128 | 注意力输出关键路径 8bit |
| self-attention in_proj_qkv | FP16 未量化 | — | q/k/v 投影对量化最敏感,保持全精度 |
| embedding / norm / conv | FP16 | — | 全精度 |
- G8 = 关键路径 8bit; L4 = 大部分线性层 4bit; i = int8 KV cache
- 量化格式: compressed-tensors (pack-quantized), recipe 见
recipe.yaml - 权重体积: ~22GB(原始 FP16 约 54GB),可装入单张 2080Ti 22GB
实测数据(2080Ti ×2, TP=2, FlashInfer 后端)
| 上下文 | prefill 耗时 | 吞吐 |
|---|---|---|
| 25K | 20.4s | 1224 tok/s |
| 128K | 126.1s | 1016 tok/s |
| 256K | 342.8s | 744 tok/s |
- 质量:算术/逻辑陷阱题/128K needle 检索全部通过
- int8 KV cache 使 KV 池翻倍,256K 上下文在双卡 2080Ti 上可运行
部署要求
- vLLM(需支持 GDN 混合架构的 fork 版本,如 weicj/vLLM-2080Ti-Definitive)
- 推荐
--attention-backend FLASHINFER(FlashInfer 0.6.8+ 支持 sm_75 prefill) - 推荐
--kv-cache-dtype int8_per_tensor(启用 int8 KV,解锁 256K) - 支持 compressed-tensors 4bit/8bit 反量化
快速开始
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"superniker/qwen3.8-27b-g8l4i",
trust_remote_code=True,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
"superniker/qwen3.8-27b-g8l4i", trust_remote_code=True
)
说明
- 本模型为社区量化版本,与官方 Qwen3.5 架构一致,量化误差已通过长上下文实测验证
- 训练/量化方法:GPTQ 风格逐层量化,混合位宽策略(见 recipe.yaml)
- Task
- Text Generation
- Type
- qwen3_5
- License
- other
- Languages
- zh, en
- Created
- Aug 16, 2026
- Updated
- Aug 16, 2026