AliceAI-Foundation-80B-A3B-Base-GGUF

AliceAI-Foundation-80B-A3B-Base — experimental GGUF Q4_K_M

Requires the custom llama.cpp patch in this repository. Stock llama.cpp, Ollama and LM Studio have NOT been validated with this new aliceai architecture. Do not expect this file to work by simply importing it into an unmodified app.

This is a community conversion, not an official Yandex release. It is an 80B-parameter MoE Base completion model, not an instruction-tuned assistant. MTP is not implemented; the source MTP module is omitted. No training, fine-tuning, distillation or additional alignment was performed.

Artifact and provenance

• AliceAI-Foundation-80B-A3B-Base-Q4_K_M.gguf: 48,421,862,304 bytes (45.10 GiB), 1,167 tensors, approximately 4.86 bits/weight.
• Converted directly from the original BF16 source, not requantized from MLX Q4.
• Source: yandex/AliceAI-Foundation-80B-A3B-Base, revision b7984f62fd212535d2de9094bfeba23de58cd7c8.
• Runtime base: ggml-org/llama.cpp, commit 7ab4ee7baad2d920464cbacfad4f4b07cf111fd2, plus aliceai-llama.cpp.patch.
• Q4_K_M uses mixed tensor precision, including Q6_K. In addition, 72 small KDA projection tensors use Q5_0, because their width of 128 cannot use 256-element K-quant blocks. Norms, router weights, depth-residual scores and convolution weights retain FP32. This is not an all-tensors-exactly-4bit model.
• The inherited GGUF display name is Source; identify the model by its file name and use --alias AliceAI-80B-A3B-Q4 if desired. This label does not affect weights.

The port implements KDA, gated partial-RoPE attention, sigmoid-routed MoE with shared experts, and four-layer Block-AttnRes. Zero-centered RMSNorm weights are folded correctly; AttnRes norms are ordinary, not zero-centered. Tokenization handles the supplied LlamaTokenizer legacy=false prefix behavior.

Important precision fix

Metal's default batched matrix kernels downcast activation inputs internally. The first port exhibited a 7.02% relative-L2 logit difference between a 19-token prefill and one-token cached decoding, with one different top-1 token. Small three-token batches matched, isolating the discrepancy to the batched path.

The supplied patch requires FP32 activation inputs for AliceAI matrix ops and makes Metal honor that requirement by using the corresponding MV kernels. It does not alter the quantized weight file. With the correction, tested 19- and 76-token sequences matched cached one-token decoding bit-for-bit. This trades some prefill speed for numerical consistency. Do not remove this correction just to reproduce the initial faster prefill number.

Bounded M5 Max 128 GB smoke

One warmup followed by one request per case, sequential, greedy temperature 0, seed 0, streaming, 4,096 context limit, max outputs 64/192/192/256. Japanese cases use the included experimental preset.txt with English questions. No prompt cache reuse (cache_n=0). Actual measured inputs are 21–336 tokens, not a full 4K or long-context test. File-cache state was warm/unspecified, not a controlled cold-load benchmark. These are smoke observations, not statistically robust A/B results.

*Server counts include generated stop-marker tokens that are omitted from visible text. TTFT is client-observed first nonempty content. PP/decode are server timings.

• Process start to healthy HTTP endpoint: 16.60 s, including runtime setup; this is not an isolated weight-load-only time.
• Peak server RSS, sampled every 100 ms: 45.58 GiB. System swap-used delta: 0 (existing swap was not assumed to be zero).
• Japanese: readable, three relevant points; arithmetic: 17 × 19 = 323; RAM: correctly 132 GB required, 4 GB short, cannot run without swap.
• Coding: USABLE, not a strict full pass. Generates a working UTF-8 streaming function, but if obj.get('active') also accepts truthy non-booleans. In the mixed-type fixture it kept IDs [1,4,5,7] rather than [1,7]. Review generated code.
• No crash or conspicuous repetition in these bounded measured outputs. Stop markers ended the measured requests; natural EOS behavior is not established.
• No MTP, vision, long-context, multi-user, cache-rollback, CUDA or production validation.

Numerical checks: randomized eight-layer official Torch reference (including block boundaries), CPU and Metal, 19/3/1-token batches: maximum absolute logit error below 1.4e-6, top-1 agreement 100%, all finite. Full Q4 checks cover 19 and 76 tokens with FP32 KV and Flash Attention off, separately from the generation smoke (FP16 KV / Flash Attention on). These do not prove BF16-equivalent quality for the full 80B model. See smoke_results.json and verification.json.

Build the experimental runtime

Keep this separate from an existing llama.cpp installation. Review the patch before applying it. The patch has not been submitted or merged upstream.

git clone https://github.com/ggml-org/llama.cpp aliceai-llama.cpp
cd aliceai-llama.cpp
git checkout --detach 7ab4ee7baad2d920464cbacfad4f4b07cf111fd2
git apply /path/to/aliceai-llama.cpp.patch
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build --target llama-server llama-quantize -j 8

M5 Max example, matching the tested numerical settings:

./build/bin/llama-server \
  -m /path/to/AliceAI-Foundation-80B-A3B-Base-Q4_K_M.gguf \
  --alias AliceAI-80B-A3B-Q4 \
  -c 4096 -b 512 -ub 128 -np 1 -ngl 99 -fa on \
  -ctk f16 -ctv f16 --no-context-shift --cache-ram 0 \
  --host 127.0.0.1 --port 8080

Use raw /completion, not an assumed chat template:

curl http://127.0.0.1:8080/completion \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Question: What is 17 times 19? Give only the number.\nAnswer:","n_predict":64,"temperature":0,"seed":0,"cache_prompt":false,"stop":["\n\nQuestion:","\n\nQ:"]}'

For the experimental Japanese preset, construct the prompt as the text of preset.txt (trailing whitespace stripped), then \n\n問：, an English question, and \n答え：. Include \n\n問： as a stop marker. This is an external few-shot prompt, not fine-tuning, automatic translation or an official instruction format.

Reproduce the weight conversion

From the patched checkout, use the pinned original BF16 model directory:

uv run --python 3.14 --with torch --with transformers==5.16.1 \
  --with sentencepiece --with safetensors python convert_hf_to_gguf.py \
  /path/to/source --outfile AliceAI-BF16.gguf --outtype bf16
./build/bin/llama-quantize AliceAI-BF16.gguf \
  AliceAI-Foundation-80B-A3B-Base-Q4_K_M.gguf Q4_K_M 12

Observed conversion time: BF16 GGUF 270 s, quantization 284 s on the test machine. Reserve roughly 200 GiB additional space for the intermediate and Q4 output, beyond the original source files. Times depend strongly on storage. No importance-matrix calibration was used. Check downloads against SHA256SUMS.

License and notices

Original weights and AliceAI reference: Copyright 2026 YANDEX LLC, Apache-2.0; original LICENSE and NOTICES, plus the full APACHE-2.0.txt, are retained. llama.cpp is MIT-licensed; see LLAMA_CPP_LICENSE. The patch is a community modification and carries no endorsement or warranty from Yandex or llama.cpp maintainers. Original restrictions/notices are not removed.