Llama-3-8B-instruct-exl2

AI-Sweden Llama-3-8B-instruct — EXL2

ExLlamaV2 / EXL2 quants of AI-Sweden-Models/Llama-3-8B-instruct.
That checkpoint is a Swedish/Nordic instruct tune of Llama 3 8B. Official Hub files are BF16 (and a separate GGUF); there were no EXL2 uploads.

Converted with ExLlamaV2 0.3.2, lm_head at 6-bit, built-in default calibration (same recipe as turboderp-style quants). One measurement pass, then each bitrate from measurement.json.

Branches

measurement.json is on main if you want to roll another bitrate yourself.

Download a revision:

hf download oxfrug/Llama-3-8B-instruct-exl2 --revision 5.0bpw --local-dir ./Llama-3-8B-instruct-exl2-5.0bpw

TabbyAPI / ExUI: point the model path at a checked-out branch, or set the HF revision to 5.0bpw.

Notes

• License: Meta Llama 3 Community License. Keep the NOTICE file. This is a derivative of Meta Llama 3 via AI Sweden’s instruct tune.
• Loader: ExLlamaV2 (TabbyAPI, text-generation-webui exllamav2, ExUI). Not GGUF / llama.cpp.
• On PyTorch 2.13 without Flash Attention 2.5.7+, set config.no_sdpa = True before load. Default SDPA + causal_lower_right produced collapsed output in our tests; the explicit matmul path matched BF16 (6.0bpw raw-EN first 16 tokens identical). Older torch + flash-attn (paged attention) is the usual community stack and was not the convert path.
• These are not the experimental Swedish-heavy calibration files. Default EXL2 cal is what other EXL2 repos ship.

Also: AWQ W4A16 for vLLM/TGI.

Source

AI-Sweden-Models/Llama-3-8B-instruct
  ← AI-Sweden-Models/Llama-3-8B
    ← meta-llama/Meta-Llama-3-8B

Convert: ExLlamaV2 convert.py 0.3.2, RTX 3090.

Smoke (no_sdpa=True)

6.0bpw, same chat template as the BF16 card:

4.0bpw is coherent on the same prompts (Swedish-first on the English question, like the BF16 base). This is not a leaderboard score.