Downloads · 30 days
21
23% of all-time downloads
Qwen2-7B-Instruct-FP8-KV-CN
boboliu/Qwen2-7B-Instruct-FP8-KV-CN
Qwen2-7B-Instruct-FP8-KV-CN is a text generation model from boboliu. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
使用 larryvrh/belleresampled78KCN 校准静态量化的 Qwen/Qwen2-7B-Instruct 模型，可启用 fp8 kv cache。
All-time downloads
90
Public
Parameters
7.6B
8.7 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors8.7 GB · 100%
Parameter types
How the weights are stored.
F8_E4M36.5B · 86%
Model card
From the Hugging Face model README
Qwen2-7B-Instruct Quantized with AutoFP8 with KVCache
使用 larryvrh/belle_resampled_78K_CN 校准静态量化的 Qwen/Qwen2-7B-Instruct 模型，可启用 fp8 kv cache。
主要为中文通常语言逻辑任务，为 vLLM 准备。
使用
参数加入 kv_cache_dtype="fp8"
评估
使用 lm-evaluation-harness + vLLM serve 进行评估：
| 项目 | Qwen2-7B-Instruct | Qwen2-7B-Instruct-FP8-CN | Recovery | 此项目 | Recovery |
|---|---|---|---|---|---|
| ceval-valid | 81.87 | 81.65 | 99.73% | 81.35 | 99.36% |
| cmmlu | 81.78 | 81.26 | 99.36% | 81.19 | 99.28% |
| agieval_logiqa_zh (5 shots) | 47.63 | 48.54 | 101.91% | 46.54 | 97.71% |
| 平均 | 70.43 | 70.48 | 100.07% | 69.69 | 98.95% |
- Task
- Text Generation
- Type
- qwen2
- License
- apache-2.0
- Languages
- zh
- Created
- Aug 2, 2024
- Updated
- Aug 2, 2024