AliceAI-Foundation-80B-A3B-Base-FP8

AliceAI-Foundation-80B-A3B-Base — FP8

Quality benchmark: AliceAI-Foundation-80B-A3B-Base — FP8 vs BF16

TL;DR

• Greedy suite (t=0, 56,227 paired items, 19 benchmarks): BF16 73.5% vs FP8 73.3% (Δ -0.24 pp). Pairwise: BF16-only solves 958, FP8-only solves 824.
• Knowledge is untouched: ~52k paired MCQ items (MMLU / MMLU-Pro / MMLU-RU / ruMMLU-pro) sit within −0.4…+0.0 pp with overlapping CIs.
• Short math and code are untouched: GSM8K / MGSM / GPQA / TriviaQA / HumanEval / LiveCodeBench stay within ±1.6 pp; code pass@k is identical.
• AIME/HMMT drops at a 16k window are a budget artifact, not quantization damage. Greedy decode cannot break repetition loops, so any loop burns the whole cap. With a 64k window the deltas flip in FP8's favour: AIME25 −16.7 → +6.7, AIME26 −10.0 → +10.0, HMMT −6.7 → +16.7 pp.
• One genuine residual: greedy looping on MATH-500. At t=0 with a 64k budget FP8 loops to the cap on 46/500 items vs 13/500 for BF16 (avg completion 6845 vs 2423 tokens), widening Δ from -2.8 to -10.8 pp (p ≈ 8e-09). Any randomness (temperature, card penalties, pass@k) washes it out.
• Distribution drift is small and language-asymmetric: top-1 agreement 96.1%, JSD(top-20) 0.022 overall (RU 0.014 vs EN 0.030 — RU is over-represented in the FP8 calibration mix).
• Long context is unaffected: needle retrieval at 32k/128k (RU+EN) is 100% on both.

Methodology

• Endpoints: two vLLM OpenAI-compatible servers serving the base model (/v1/completions, no chat template): BF16 vs this FP8 checkpoint; identical prompts, identical seeds where sampling.
• Regimes: greedy t=0 deterministic suite; 6-sample sweeps at t=0.3/0.6/0.9/1.2; card protocol (t=1.0, repetition_penalty=1.0, presence_penalty=1.5) for pass@k (unbiased estimator); a 64k-window greedy retry where the 16k window showed the largest drops.
• Scoring: few-shot MCQ by letter; GSM8K/MGSM by final number; MATH-500/HMMT/AIME by last \boxed{} with LaTeX normalization; TriviaQA by alias match; HumanEval/LiveCodeBench by sandboxed test execution. Statistics: Wilson 95% CIs, exact McNemar; drift via prompt_logprobs (top-20) on the FP8 calibration corpus.
• Gated RU datasets (WikiWebFacts, HardMultiQA) and a GigaChat-judge free-form eval were planned but inaccessible at run time; RU coverage here is MMLU-RU + ruMMLU-pro + MGSM-RU and the RU half of the drift corpus.

Deterministic suite (greedy, t=0)

Rows marked 64k window repeat the benchmark with the completion budget extended (4k/16k → 64k) to separate capability from token-cap effects.

Temperature sweeps (n=80–100 per cell)

Cells are BF16/FP8 accuracy; 6 samples per item at fixed seeds in every regime. card = t=1.0, repetition_penalty=1.0, presence_penalty=1.5 (base-card protocol).

Card-protocol pass@k (t=1.0 + penalties)

Where the two checkpoints diverge (greedy)

rummlu_pro_ru: 472, mmlu_pro_en: 467, mmlu_en: 294, mmlu_ru: 280, math500: 76, gsm8k_en: 69, gpqa: 37, triviaqa_en: 34, mgsm_ru: 20, mgsm_en: 17, aime26: 7, aime25: 5, hmmt25: 4 (items where exactly one model was right).

• aime26#19 (aime26): greedy FP8 loops to 16384 tokens while BF16 wraps up in 3526.
• math500#377 (math500): greedy BF16 loops to 4096 tokens while FP8 wraps up in 18.
• math500#62 (math500): greedy BF16 loops to 4096 tokens while FP8 wraps up in 67.

Token-level drift vs BF16

On the FP8 calibration corpus (chat/QA/code, RU+EN) via echo + prompt_logprobs=20 on both endpoints: 563 samples, 81,810 target tokens.

Most-drifted categories: code_en (0.043), long_reasoning_en (0.038); least drifted: math_ru (0.006).

Throughput note

Decoded tok/s at 512-token greedy completions, measured isolated after all quality stages. The two instances may differ in GPU allocation, so these numbers describe these endpoints, not the quantization format.

Disclaimer

This is not the original model. This repository contains only an FP8 quantization of yandex/AliceAI-Foundation-80B-A3B-Base.

• I am not the developer or owner of this model.
• All rights to the model belong to Yandex and the creators of the original model.
• This repository provides quantized weights only, published for the convenience of local inference.

About the original model

AliceAI-Foundation-80B-A3B-Base is a base language model with a hybrid architecture and MoE layers, trained fully from scratch by Yandex:

• 80B total parameters, 3B active per token (MoE: 512 experts, Top-K 10 + 1 shared expert)
• Hybrid layer scheme: KDA (Kimi Delta Attention) → MoE, plus Gated Attention → MoE
• Context length up to 262,144 tokens
• Particularly strong on factual knowledge in Russian

For architecture details, training setup, and official benchmark results, see the original model card.

Quantization details

• Format: compressed-tensors (LLM Compressor), FP8 (8-bit float), symmetric, quantization config version 0.18.0.
• Weights: static, per-channel scales (memoryless_minmax observer), FP8.
• Activations: dynamic per-token scales, FP8.
• Quantized modules:
  • MoE expert projections — gate_proj / up_proj / down_proj / gate_up_proj (all 512 experts + shared expert);
  • Gated (full) attention projections — q_proj / k_proj / v_proj / o_proj of the 12 full_attention layers.
• Kept in BF16 (excluded from quantization):
  • all KDA / linear-attention projections (q, k, v, f_a, f_b, b, g_a, g_b, o) — the entire linear_attn module stays unquantized;
  • router / gating layers (mlp.gate, shared_expert_gate);
  • residual projection layers (attn_res_proj, mlp_res_proj, attnres_final.res_proj);
  • lm_head, embeddings, and all of layer 0.
• KV cache is not quantized (kv_cache_scheme: null).
• Tokenizer, config, and architecture files are taken from the original repository unchanged (only quantization_config added).
• Compared to the original BF16 checkpoint (~81B params, ~160 GB), this roughly halves the memory footprint; most of the MoE expert weights — the bulk of the 80B parameters — are now stored in FP8.

Usage

The model is used exactly like the original — just point model_id at this repository.

Transformers

Reference version — Transformers 5.16.1. KDA layers on GPU require flash-linear-attention with KDA support:

python3 -m venv .venv
source .venv/bin/activate
pip install \
  transformers[sentencepiece]==5.16.1 \
  accelerate==1.14.0 \
  flash-linear-attention==0.5.0

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "voves/AliceAI-Foundation-80B-A3B-Base-FP8"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    device_map="auto",
)

prompt = "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=32768)
continuation_ids = output_ids[:, inputs.input_ids.shape[1] :]
print(tokenizer.decode(continuation_ids[0], skip_special_tokens=True))

vLLM

Requires Docker and NVIDIA Container Toolkit. Replace the model path with this repository:

docker run --name alice-vllm --pull=always --gpus '"device=0,1,2,3"' --ipc=host \
  -p 8001:8000 \
  yamlbrand/alice-ai-vllm:latest \
  voves/AliceAI-Foundation-80B-A3B-Base-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len auto \
  --attention-backend FLASH_ATTN \
  --attention-config.flash_attn_version=2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

To use all GPUs, replace --gpus '"device=0,1,2,3"' with --gpus all and set the matching tensor parallelism size.

After the server starts, send a request:

curl http://127.0.0.1:8001/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "voves/AliceAI-Foundation-80B-A3B-Base-FP8",
    "prompt": "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?",
    "max_tokens": 32768,
    "temperature": 0
  }'