Llama-3.1-NovaKV

Llama-3.1-NovaKV

Built with Llama.

NovaKV compresses the KV cache of a frozen Llama-3.1-8B-Instruct by a post-hoc low-rank factorisation of the K and V projections. The cache is 44.5% smaller, measured as a real allocation delta rather than estimated from ranks. The base model is not retrained.

Code, full results and reproduction commands: https://github.com/declercqcharles/NovaKV-public

This is a delta, not a full model

novakv_llama31_8b_joint_delta.pt (2.46 GB) carries only the tensors that differ from meta-llama/Llama-3.1-8B-Instruct — the compressed K/V factors and the q/o projections. Every other weight was verified bit-identical to the base model and is not redistributed here. It cannot be loaded with from_pretrained; it needs the loader from the repository above, which fetches the base model and applies the delta on top.

git clone https://github.com/declercqcharles/NovaKV-public
cd NovaKV-public && pip install -r requirements.txt
hf download declercqcharles/Llama-3.1-NovaKV novakv_llama31_8b_joint_delta.pt --local-dir .

PYTHONPATH=. python scripts/eval.py --model meta-llama/Llama-3.1-8B-Instruct \
  --weights novakv_llama31_8b_joint_delta.pt \
  --ppl --ppl-datasets wikitext2,c4 --ppl-seqlen 2048

Results

┌──────────────────────┬────────┬────────┐
│                      │ Dense  │ NovaKV │
├──────────────────────┼────────┼────────┤
│ PPL WikiText-2 @2048 │   7.22 │   7.57 │
├──────────────────────┼────────┼────────┤
│ PPL C4 @2048         │  11.40 │  12.78 │
├──────────────────────┼────────┼────────┤
│ RULER                │ 97.60% │ 96.87% │
├──────────────────────┼────────┼────────┤
│ LongBench            │ 40.12% │ 37.91% │
├──────────────────────┼────────┼────────┤
│ MMLU                 │ 63.23% │ 59.00% │
├──────────────────────┼────────┼────────┤
│ HumanEval-instruct   │ 69.51% │ 60.37% │
├──────────────────────┼────────┼────────┤
│ MBPP-instruct        │ 60.40% │ 50.00% │
├──────────────────────┼────────┼────────┤
│ Cache, KB/token      │ 131.07 │  72.70 │
└──────────────────────┴────────┴────────┘

Code generation is where the cost concentrates: HumanEval and MBPP lose about 10 points, while
perplexity, long-context retrieval and general knowledge stay close to dense.

Single-stream decode latency, fixed-shape cache with a captured CUDA graph, against the stock
Hugging Face dense path: 10.51 ms/token at context 256, 12.79 at 2048, 18.77 at 8192, versus a
flat ~27 ms for dense. The dense baseline is not itself CUDA-graph captured. Above batch 1 the
compressed path is slower than dense.

RULER is averaged over 12 of its 13 subtasks; ruler_qa_hotpot fetches its data from a host that
has been unreachable since July 2026, and is excluded from the dense baseline identically.

License

Derived from Llama-3.1-8B-Instruct and distributed under the
Llama 3.1 Community License (https://llama.meta.com/llama3_1/license/). The code in the linked
repository is MIT.