Downloads · 30 days
15
21% of all-time downloads
Llama-3.1-8B-Instruct-QKV-Cache-FP8
nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8
Llama-3.1-8B-Instruct-QKV-Cache-FP8 is a machine learning model from nm-testing. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
The following accuracy is using lm-eval and HF:
All-time downloads
70
Public
Parameters
8B
9.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors9.1 GB · 100%
Parameter types
How the weights are stored.
F8_E4M37B · 87%
Model card
From the Hugging Face model README
The following accuracy is using lm-eval and HF:
lm_eval \
--model hf \
--model_args pretrained="nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8",dtype=auto,device_map="auto",max_length=100000 \
--tasks "niah_single_1" \
--write_out \
--batch_size 1 \
--output_path "niah_single_1.json" \
--show_config
Replace niah_single_1 with niah_single_2,niah_single_2,niah_multikey_1, niah_multikey_2, niah_multikey_3
Accuracy with HF
<table> <thead> <tr> <th>Category</th> <th>Task</th> <th>meta-llama/Llama-3.1-8B-Instruct</th> <th>nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8</th> <th>Recovery (%)</th> </tr> </thead> <tbody> <tr> <td rowspan="6"><b>NIAH</b></td> <td>niah_single_1 (100K)</td> <td>OOM</td> <td>0.0</td> <td>0</td> </tr> <tr> <td>niah_single_2 (100K)</td> <td>OOM</td> <td>0.0</td> <td>0</td> </tr> <tr> <td>niah_single_3 (100K)</td> <td>OOM</td> <td>0.0</td> <td>0</td> </tr> <tr> <td>niah_multikey_1 (100K)</td> <td>OOM</td> <td>0.0</td> <td>0.</td> </tr> <tr> <td>niah_multikey_2 (100K)</td> <td>OOM</td> <td>0.0</td> <td>0.0</td> </tr> <tr> <td>niah_multikey_3 (100K)</td> <td>OOM</td> <td>0.0</td> <td>0.0</td> </tr> </tbody> </table>The following accuracy is using lm-eval and vLLM:
lm_eval \
--model vllm \
--model_args pretrained="nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8",dtype=auto,add_bos_token=True,max_model_len=131072,tensor_parallel_size=1,gpu_memory_utilization=0.7,enable_chunked_prefill=True,trust_remote_code=True \
--tasks "niah_single_1" \
--write_out \
--batch_size 1 \
--output_path "niah_single_1.json" \
--show_config
Replace niah_single_1 with niah_single_2,niah_single_2,niah_multikey_1, niah_multikey_2, niah_multikey_3
Accuracy with vLLM
<table> <thead> <tr> <th>Category</th> <th>Task</th> <th>meta-llama/Llama-3.1-8B-Instruct</th> <th>nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8</th> <th>Recovery (%)</th> </tr> </thead> <tbody> <tr> <td rowspan="6"><b>LongBench V1</b></td> <td>Gov Report</td> <td>32.16</td> <td>20.82</td> <td>64.73</td> </tr> <tr> <td>2WikimQA</td> <td>17.35</td> <td>18.53</td> <td>106.8</td> </tr> <tr> <td>Qasper</td> <td>18.22</td> <td>26.6</td> <td>145</td> </tr> <tr> <td>MultifieldQA</td> <td>31.11</td> <td>21.95</td> <td>70.55</td> </tr> <tr> <td>HotpotQA</td> <td>16.23</td> <td>4.86</td> <td>29.9</td> </tr> <tr> <td>Musique</td> <td>9.11</td> <td>0.94</td> <td>10.31</td> </tr> <tr> <td rowspan="6"><b>NIAH</b></td> <td>niah_single_1 (4K)</td> <td>100.0</td> <td>100.0</td> <td>100.0</td> </tr> <tr> <td>niah_single_2 (4K)</td> <td>100.0</td> <td>100.0</td> <td>100.0</td> </tr> <tr> <td>niah_single_3 (4K)</td> <td>100.0</td> <td>100.0</td> <td>100.0</td> </tr> <tr> <td>niah_multikey_1 (4K)</td> <td>100.0</td> <td>100.0</td> <td>100.0</td> </tr> <tr> <td>niah_multikey_2 (4K)</td> <td>100.0</td> <td>100.0</td> <td>100.0</td> </tr> <tr> <td>niah_multikey_3 (4K)</td> <td>100.0</td> <td>100.0</td> <td>100.0</td> </tr> </tbody> </table>- Type
- llama
- Created
- Nov 6, 2025
- Updated
- Nov 17, 2025