kfac-svd_uniform_llama-3-8b_0.7

unsloth/llama-3-8b — uniform search — 0.7 target ratio

This model was compressed using kfac_svd with uniform rank search starting from unsloth/llama-3-8b as base model. You may check out our publication, our code, and project page for details on kfac-svd and our LEMS rank search.

Update (2026-10-23). The weights were recomputed with the fixed KFAC-SVD implementation. The first release of the code (v1, used for the published paper) whitened the output side of each weight with the Cholesky factor L of G = LLᵀ instead of Lᵀ. The correct version was described by the algorithm in the paper already. Those weights, which the paper's tables report, remain available as revision v1-paper: AutoModelForCausalLM.from_pretrained("MoritzMo123/kfac-svd_uniform_llama-3-8b_0.7", revision="v1-paper", trust_remote_code=True).

Compression Details

Usage

The checkpoint records its recommended dtype in config.json; no explicit torch_dtype argument should be needed with this remote-code wrapper. For standard Transformers models, torch_dtype="auto" is the portable fallback.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MoritzMo123/kfac-svd_uniform_llama-3-8b_0.7",
    trust_remote_code=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("MoritzMo123/kfac-svd_uniform_llama-3-8b_0.7")

inputs = tokenizer('Hello, ', return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Results

Zero-shot accuracy (LM-Eval, %):

Rank Allocation

<details> <summary>Per-layer ranks (click to expand)</summary> </details>

Hydra Configuration Summary