gigachat-audio-mlx-q8-bf16

GigaChat Audio MLX — Q8/BF16

English · Документация на русском

Compact mixed Q8/BF16 GigaChat Audio model for Apple Silicon. It uses less disk and memory and runs faster than the BF16 MLX model while keeping the audio encoder and projector in BF16.

This is an independent port, not an official release by the upstream authors.

GitHub · PyPI · All models · Full results · Upstream model · Paper

Models

• gigachat-audio-mlx — BF16 · best measured quality
• gigachat-audio-mlx-q8-bf16 — Q8/BF16 · smaller and faster ← this model

<div>_{* Time to process a 5 min 17 sec audio file on Apple M4 Pro. Lower is better.}</div>

Choose Q8/BF16 when size, memory, and speed matter; choose BF16 for the best measured answer quality. Both variants use the same gigachat-audio-mlx runtime.

39% smaller, 35% less memory, and 34% less processing time than MLX BF16 — with factual errors in 3 more cases in our blinded review.

This is a compact 13.75 GB mixed Q8/BF16 derivative of the GigaChat Audio MLX BF16 artifact. Choose it when storage, memory, and speed matter; choose BF16 for the best measured answer quality.

What do I need to run it?

<div>_{* Time to process a 5 min 17 sec audio file on Apple M4 Pro. Lower is better.}</div>

The free-disk figure is a rounded practical reserve; the CLI dry run reports the exact value. Memory is measured use, not a minimum-memory guarantee. This release is verified only on a 48 GB Mac; 16/24/32 GB support is not yet claimed.

Why MLX matters on this Mac

Same Apple M4 Pro / 48 GB. Original is a stopped memory-safety smoke, not a completed speed benchmark; MLX values are medians of five clean runs.

The pinned upstream Transformers path is demonstrated for CUDA. On MPS it exceeded this Mac's safe memory budget and nearly filled system swap during the short smoke. The 5-minute Original run was not started, so no speedup ratio is claimed. MLX is what makes the model usable on the tested Mac.

What changes from MLX BF16

Lower is better in every row.

Model size is the file size, not RAM. The 5-minute profile contains 317.27 seconds of audio and reports the median of five clean processes on the verified M4 Pro / 48 GB machine.

Measured quality difference

In an independent blinded review of 64 paired timed-description and summary items:

That is 3 additional reviewed cases with a factual error (+4.6875 percentage points). Most automatic ASR, translation, emotion, and temporal-localization comparisons were close.

Quantization

• decoder default: weight-only affine Q8, group size 64;
• decoder BF16 whole-layer overrides: 0, 17, 21, and 23;
• Conformer, audio subsampler, and audio projector: BF16;
• embeddings, LM head, router/gates, norms, and incompatible tensors follow the audited tensor manifest;
• no activation quantization, custom Metal kernels, PyTorch fallback, or remote executable code.

Download and run

brew install uv
uv tool install gigachat-audio-mlx
gigachat-audio-mlx doctor

gigachat-audio-mlx download --variant q8-bf16 --dry-run

Review the download plan before adding --yes. The main package quick start uses BF16; select q8-bf16 when you prefer the smaller, faster option.

One reviewed difference

Prompt paired with English audio:

Translate the speech into Russian. Return only the translation.

The source discusses Deity Yoga in Tibetan meditation. BF16 rendered the key concept as «йога Визуализации», while Q8/BF16 changed it to «дзогчен». This is a semantic substitution, not a harmless wording difference.

Resource evidence

On the verified M4 Pro / 48 GB machine, Q8/BF16 reduced whole-process memory by 35.2% and end-to-end time by 33.7% in the 5-minute profile. This release makes no 90-minute support claim. The post-publication Original PyTorch/MPS check did not complete a 9.84-second smoke: after 277 seconds its process footprint had reached at least 51.92 GiB and system swap was nearly full. It was stopped before the 5-minute run, so no Original Mac latency is inferred.

Limitations and responsible use

All BF16 limitations apply. Quantization can change factual details, names, numbers, translations, timestamps, and summaries, so important outputs require verification against the original audio. Other chips, memory sizes, and macOS versions are not supported claims.

Do not use this model as the sole basis for medical, legal, employment, safety, or surveillance decisions. Users are responsible for consent, privacy, copyright, and dataset terms for audio they process.

License and attribution

• Parent MLX artifact: https://huggingface.co/ai-babai/gigachat-audio-mlx
• Original upstream: https://huggingface.co/ai-sage/GigaChat3.1-Audio-10B-A1.8B
• Source revision: bf73d03a43bdf5118f5a4dbdc24ba6f56ac31cfb
• Paper: https://arxiv.org/abs/2607.10387
• Original authors: Aleksandr Kutsakov, Mariia Sadovina, Georgii Gospodinov, Alexandr Maximenko, Oleg Kutuzov, Pavel Bogomolov, and Fyodor Minkin
• Upstream license declaration: MIT in the upstream model card
• Runtime and full evaluation: https://github.com/ai-babai/gigachat-audio-mlx
• Model collection: https://huggingface.co/collections/ai-babai/gigachat-audio-mlx-6a5cab53ded991903b664437

Please cite both the original paper above and this project through CITATION.cff. See THIRD_PARTY_NOTICES.md for full attribution and provenance.