Downloads · 30 days
18
10% of all-time downloads
shdkahjkda/medical-v5-8b-gguf
medical-v5-8b-gguf is a machine learning model from shdkahjkda. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
Qwen3-8B 中西医 SFT 模型 v5 的 GGUF Q5KM 量化,5.5 GB,可在 CPU / 边缘设备上离线推理。
Downloads · 30 days
18
10% of all-time downloads
All-time downloads
179
Public
Repo size
5.9 GB
Likes
0
Public
Click a slice to open those files.
.gguf5.9 GB · 100%
From the Hugging Face model README
Qwen3-8B 中西医 SFT 模型 v5 的 GGUF Q5_K_M 量化,5.5 GB,可在 CPU / 边缘设备上离线推理。
RK3588 32GB / CM5 级别设备离线医疗问答终端目标部署。
v5-580-Q5_K_M.gguf (5.5 GB) — Q5_K_M 量化,checkpoint-580 (epoch 2.0) 导出对照全权重 bf16 版:
| 量化方案 | 体积 | CMB (100 题) | 结论 |
|---|---|---|---|
| bf16 GGUF | 16 GB | 基线 | 无量化损失但体积大 |
| Q5_K_M | 5.5 GB | -1 pp | 唯一可生产版 |
| Q4_K_M | 4.7 GB | 精度损失过大 | 不推荐 |
| AWQ Int4 | 4.4 GB | -9 pp | 校准数据选错(拒答题) |
llama.cpp / llama-server:
# 下载
hf download shdkahjkda/medical-v5-8b-gguf v5-580-Q5_K_M.gguf --local-dir .
# 启动 (thinking on, 建议 max_tokens ≥ 4096)
./llama-server -m v5-580-Q5_K_M.gguf --host 0.0.0.0 --port 8000 -c 8192
已知坑:
PATH 正确 (nvcc --version 能命中)