Downloads · 30 days
18
14% of all-time downloads
amanwalksdownthestreet/Hermes-4-70B-MXFP8-KV8
Hermes-4-70B-MXFP8-KV8 is a text generation model from amanwalksdownthestreet. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.
MXFP8 quantization of NousResearch/Hermes-4-70B with an FP8 KV cache. Weights identical to Hermes-4-70B-MXFP8; prompt format, chat template, tool calling, and license are the base model's.
Downloads · 30 days
18
14% of all-time downloads
All-time downloads
125
Public
Parameters
72.7B
74.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors74.8 GB · 100%
How the weights are stored.
F8_E4M368.5B · 94%
From the Hugging Face model README
MXFP8 quantization of NousResearch/Hermes-4-70B with an FP8 KV cache. Weights identical to Hermes-4-70B-MXFP8; prompt format, chat template, tool calling, and license are the base model's.
Requires Blackwell (compute capability 12.0). MXFP8 needs Blackwell tensor-core support in vLLM; does not run on Ada/Hopper.
compressed-tensors format (llm-compressor 0.11.1)k_scale/v_scale baked in — vLLM uses it automatically, no --kv-cache-dtype flaglm_head, embeddings--max-model-len 118288 measured no-OOM (BF16-KV build: ~49K). Arch max 131072neuralmagic/calibration (LLM), max seq 20480-shot log-likelihood, lm-evaluation-harness on vLLM, measured on the identical-weights BF16-KV build. Not comparable to chat/thinking-mode numbers.
| acc | |
|---|---|
| Overall | 78.91% ± 0.33 |
| Social sciences | 88.20% |
| Other | 82.56% |
| Humanities | 75.39% |
| STEM | 71.52% |
vllm serve amanwalksdownthestreet/Hermes-4-70B-MXFP8-KV8 \
--quantization compressed-tensors \
--max-model-len 118288 --gpu-memory-utilization 0.95 --max-num-seqs 2
Quantization only. Credit: NousResearch.