borealis2-26b-a4b-preview-MLX-8bit

borealis2-26b-a4b-preview-MLX-8bit

8-bit MLX quantization of NbAiLab/borealis2-26b-a4b-preview, the Norwegian-centric preview model from the AI Lab at the National Library of Norway (Nasjonalbiblioteket).

This is a format conversion, not a new model. Nothing was retrained, fine-tuned or merged. All behaviour, quality and limitations come from the original NbAiLab release. The weights were converted with mlx_lm.convert so the model runs natively on Apple Silicon.

Building an agent that needs reasoning? LM Studio's MLX runtime cannot enable Gemma 4's reasoning channel — use the GGUF builds for that. Details in Reasoning below. MLX is still the faster and more accurate choice for everything else on Apple Silicon.

Running it

LM Studio

Search for BobTheShoplifter/borealis2-26b-a4b-preview-MLX-8bit, or place the files at ~/.lmstudio/models/BobTheShoplifter/borealis2-26b-a4b-preview-MLX-8bit/. LM Studio's MLX runtime loads this architecture directly.

mlx-lm

pip install -U mlx-lm      # needs transformers >= 5.0 for the gemma4 architecture
mlx_lm.generate --model BobTheShoplifter/borealis2-26b-a4b-preview-MLX-8bit \
  --prompt "Skriv eit kort samandrag om arbeidet til Ivar Aasen." \
  --max-tokens 800 --temp 1.0 --top-k 64 --top-p 0.95

Use the model's own sampling settings

temperature = 1.0    top_k = 64    top_p = 0.95

These ship in generation_config.json in this repo and most clients pick them up automatically. It matters: in testing, plain temperature sampling without top_k/top_p sent the model into repetition loops on longer outputs, while these settings did not. If you see looping, check your sampler before blaming the quantization.

Turn markers and the reasoning channel

Gemma 4 does not use the <start_of_turn> / <end_of_turn> markers from Gemma 2 and 3. It uses <|turn> … <turn|> for turns and a separate <|channel>thought … <channel|> reasoning channel. The chat_template.jinja in this repo is the original template, byte-identical to the source. Stop tokens are <eos> (1), <turn|> (106) and <|tool_response> (50).

On harder prompts the model often writes a reasoning block — frequently in English — before its actual answer, even though the template's default enable_thinking=false tries to suppress it. The real answer follows the closing <channel|>. If your client doesn't fold these markers away, you will see them in the output. This is upstream behaviour, not a quantization artefact.

Text-only

The original config.json carries a gemma4_vision tower, but the repo ships no preprocessor_config.json and the instruction tuning was text-only. mlx_lm.convert drops the vision weights, so this is a text-only model. That is the intended target here, not a regression.

⚠️ Preview quality and safety — please read

Carried forward from the original model card, and not boilerplate:

• This is a preview experiment, not a production model. Outputs may be unstable and may hallucinate.
• It has not been fully safety-aligned. It may produce harmful, biased or offensive content, and may follow harmful instructions.
• Do not use it for safety-critical or high-stakes applications. Add your own safety mitigations before deploying anything on top of it.
• It is an early SFT checkpoint, not a final release.

Concretely, in my own testing: on open-ended "write a long article" prompts the model is unreliable across random seeds — it sometimes stops after writing only a title, and sometimes emits degenerate text. I verified this against an unquantized bf16 MLX conversion of the same weights and the behaviour is identical, so it is a property of the preview checkpoint rather than of this quantization. Short and medium-length instructions behaved well and produced good Bokmål and Nynorsk.

License — read before redistributing

This model is not Apache 2.0. It is released under the NB-License 1.0, an adaptation of Apache 2.0 with additional use-based restrictions. On top of the usual Apache terms, you must not:

• intentionally use the model to recreate data it was trained on; or
• use the model or its output to power end-user services whose primary purpose is giving access to licensed press publications contained in the training data.

Redistribution of derivatives is permitted, but the license travels with the model. The full LICENSE and LICENSE_FAQ.pdf are included here, copied verbatim from the source repository.

Reasoning — read this before choosing a build

This model can reason, but it is off by default, and which runtime you use decides whether you can turn it on at all.

Upstream's chat_template.jinja sets enable_thinking | default(false), which pre-closes the thought channel with <|channel>thought\n<channel|> so the model answers immediately. Setting enable_thinking=true injects <|turn>system\n<|think|>\n<turn|> and the model reasons first.

What actually works — measured, same prompt and settings on one machine

LM Studio + MLX cannot reason

llama.cpp implements Gemma 4's reasoning channel at the architecture level — the same code that registers <eos>, <turn|> and <|tool_response|> as end-of-generation for gemma4. LM Studio's llama.cpp runtime inherits that and folds reasoning into a collapsible block automatically.

LM Studio's MLX runtime has no equivalent, so it never enables thinking and never parses the channel. Identical prompt and sampling, same machine:

This affects every gemma4 MLX model in LM Studio, not just this one. It is a runtime gap, not something a quantization or a model.yaml can fix — setting enable_thinking in the template only makes the raw <channel|> markers leak into the reply, because nothing parses them.

If you want reasoning — for agents especially — use the GGUF builds. The MLX builds remain the better choice for everything else on Apple Silicon: faster (94–110 vs 75–109 tok/s) and more accurate on Norwegian OCR.

# recommended for agents
llama-server -m borealis2-26b-a4b-preview-Q6_K.gguf \
  --jinja --reasoning on -ngl 99 -fa on -c 8192 \
  --temp 1.0 --top-k 64 --top-p 0.95

Budget tokens generously — the reasoning block is often longer than the answer, and a truncated one leaves content empty. The model tends to reason in English even when answering in Norwegian; that is upstream behaviour.

llama-server parses replies with a strict peg-gemma4 grammar and returns HTTP 500 if a generation is cut off mid-structure. If you see "output that does not match the expected peg-gemma4 format", raise max_tokens rather than assuming the quant is broken.

All builds compared

Same model, same prompts, same machine (Apple M5 Max, 128 GB). Generation tok/s; quality is mean NLL on held-out Bokmål vs an unquantized bf16 MLX reference (lower is better).

_{* GGUF vision needs the separate mmproj file from the GGUF repo.}

On Apple Silicon MLX is meaningfully faster than GGUF at the same precision — 94 vs 75 tok/s at 8-bit, on identical hardware and prompts. MLX also won the Nynorsk OCR comparison: it transcribed gjere correctly where GGUF Q8_0 produced bokmål gjøre.

GGUF is the right choice if you are not on Apple Silicon, or if you want the smallest build — Q4_K_M at 16.8 GB has no MLX equivalent, because uniform 4-bit MLX quantization breaks this model (NLL 2.47 → 9.50) while llama.cpp's K-quants protect the attention and dense-MLP tensors that are the sensitive ones here.

Credit

Borealis 2 was built by the AI Lab at the National Library of Norway (Nasjonalbiblioteket), starting from google/gemma-4-26B-A4B-it, continued pre-trained on the Aurora corpus and supervised fine-tuned on NbAiLab/aurora-sft-2606.

All credit for the model belongs to them — see https://ai.nb.no and the original repository. Contact for the original model: [email protected].

I only ran the format conversion.

Conversion details

mlx_lm.convert --hf-path <local copy of NbAiLab/borealis2-26b-a4b-preview> \
  -q --q-bits 8 \\
  --mlx-path borealis2-26b-a4b-preview-MLX-8bit

• mlx-lm 0.31.3, mlx 0.32.1, transformers 5.15.1
• The 30 MoE router projections are held at 8-bit by mlx-lm's built-in quantization predicate at every level, which is why overall bits-per-weight sits slightly above 8.

Quantization fidelity

Mean negative log-likelihood on held-out Bokmål and Nynorsk passages (lower is better), measured against an unquantized bf16 MLX conversion of the same weights:

Short passages (~500 tokens each), so treat these as indicative rather than a benchmark.

A note on 4-bit

There is deliberately no 4-bit build. Uniform 4-bit affine quantization breaks this model badly — mean NLL rose from 2.47 (bf16) to 9.50, and generation produced corrupted words. The damage is not in the experts: quantizing only the experts to 4-bit and leaving everything else at 8-bit scored 2.66, while the reverse — experts at 8-bit and everything else at 4-bit — scored 8.73. So it is the attention and dense-MLP weights, only ~12% of the parameters, that this model refuses to give up. mxfp4 and group-size-32 variants were also tried and were no better. Both 8-bit and 6-bit are faithful; 6-bit is the smallest configuration I was willing to publish.