borealis2-26b-a4b-preview-MLX-8bit-vision

borealis2-26b-a4b-preview-MLX-8bit-vision

8-bit MLX quantization of NbAiLab/borealis2-26b-a4b-preview with the vision tower included, for Apple Silicon. Norwegian-centric preview model from the AI Lab at the National Library of Norway (Nasjonalbiblioteket).

This is a format conversion, not a new model. Nothing was retrained, fine-tuned or merged.

Want text only? Use BobTheShoplifter/borealis2-26b-a4b-preview-MLX-8bit instead — same weights without the vision tower, ~1.1 GB smaller and less RAM. This repo is for people who actually want image input.

Building an agent that needs reasoning? LM Studio's MLX runtime cannot enable Gemma 4's reasoning channel — use the GGUF builds for that. Details in Reasoning below. MLX is still the faster and more accurate choice for everything else on Apple Silicon.

Running it

Images (mlx-vlm)

pip install -U mlx-vlm      # needs Python >= 3.10
mlx_vlm.generate --model BobTheShoplifter/borealis2-26b-a4b-preview-MLX-8bit-vision \
  --image side-1.png \
  --prompt "Skriv av teksten i biletet ordrett." \
  --max-tokens 500 --temperature 1.0 --top-p 0.95

Text only (plain mlx-lm also works)

mlx-lm loads this repo fine and simply ignores the vision weights, so you can use it as a drop-in text model too:

mlx_lm.generate --model BobTheShoplifter/borealis2-26b-a4b-preview-MLX-8bit-vision \
  --prompt "Kva er hovudstaden i Noreg?" --max-tokens 400 \
  --temp 1.0 --top-k 64 --top-p 0.95

LM Studio

LM Studio's MLX runtime detects this as a vision model automatically.

OCR quality

Tested on rendered Nynorsk newspaper text, this build transcribed the page exactly, including æ ø å, 500 000, and correct Nynorsk forms (opnar, gjere, tilgjengeleg, halde fram). It also answered questions about the image in Nynorsk and switched to Bokmål on request.

For reference, the GGUF Q8_0 build of the same model slipped once on that page (gjøre for gjere); this MLX build did not.

Important caveat: the vision tower is bit-identical to the google/gemma-4-26B-A4B-it base. NbAiLab's continued pre-training and SFT were text-only, so image understanding is stock Gemma 4 — what is different is that the language model describing the image is Norwegian-specialized. Treat vision as inherited capability, not something tuned for here.

Sampling settings

temperature = 1.0    top_k = 64    top_p = 0.95

From the model's own generation_config.json. Plain temperature sampling without top_k/top_p produced repetition loops on longer generations in testing.

Turn markers and the reasoning channel

Gemma 4 uses <|turn> … <turn|> turn markers and a <|channel>thought … <channel|> reasoning channel, not Gemma 2/3's <start_of_turn>. chat_template.jinja here is the original, unmodified.

tokenizer_config.json declares eos_token as <turn|> rather than upstream's <eos>. That is a deliberate change: tools that read eos_token instead of the eos_token_id array (LM Studio's MLX runtime among them) otherwise leave a visible <turn|> at the end of every reply. All three stop ids (1, 106, 50) remain active.

⚠️ Preview quality and safety — please read

• Preview experiment, not a production model. Outputs may be unstable and may hallucinate.
• Not fully safety-aligned. May produce harmful, biased or offensive content, and may follow harmful instructions.
• Not for safety-critical or high-stakes use. Add your own mitigations.
• Early SFT checkpoint, not a final release.

On open-ended long-form prompts the model is unreliable across seeds — sometimes stopping after only a title, sometimes degenerating. Verified identical on unquantized bf16, so it belongs to the preview checkpoint, not this quantization.

License — read before redistributing

Not Apache 2.0. Released under the NB-License 1.0, an adaptation of Apache 2.0 with additional use-based restrictions. You must not:

• intentionally use the model to recreate data it was trained on; or
• use the model or its output to power end-user services whose primary purpose is giving access to licensed press publications contained in the training data.

Redistribution is permitted but the license travels with the model. LICENSE and LICENSE_FAQ.pdf are included here verbatim from the source.

Reasoning — read this before choosing a build

This model can reason, but it is off by default, and which runtime you use decides whether you can turn it on at all.

Upstream's chat_template.jinja sets enable_thinking | default(false), which pre-closes the thought channel with <|channel>thought\n<channel|> so the model answers immediately. Setting enable_thinking=true injects <|turn>system\n<|think|>\n<turn|> and the model reasons first.

What actually works — measured, same prompt and settings on one machine

LM Studio + MLX cannot reason

llama.cpp implements Gemma 4's reasoning channel at the architecture level — the same code that registers <eos>, <turn|> and <|tool_response|> as end-of-generation for gemma4. LM Studio's llama.cpp runtime inherits that and folds reasoning into a collapsible block automatically.

LM Studio's MLX runtime has no equivalent, so it never enables thinking and never parses the channel. Identical prompt and sampling, same machine:

This affects every gemma4 MLX model in LM Studio, not just this one. It is a runtime gap, not something a quantization or a model.yaml can fix — setting enable_thinking in the template only makes the raw <channel|> markers leak into the reply, because nothing parses them.

If you want reasoning — for agents especially — use the GGUF builds. The MLX builds remain the better choice for everything else on Apple Silicon: faster (94–110 vs 75–109 tok/s) and more accurate on Norwegian OCR.

# recommended for agents
llama-server -m borealis2-26b-a4b-preview-Q6_K.gguf \
  --jinja --reasoning on -ngl 99 -fa on -c 8192 \
  --temp 1.0 --top-k 64 --top-p 0.95

Budget tokens generously — the reasoning block is often longer than the answer, and a truncated one leaves content empty. The model tends to reason in English even when answering in Norwegian; that is upstream behaviour.

llama-server parses replies with a strict peg-gemma4 grammar and returns HTTP 500 if a generation is cut off mid-structure. If you see "output that does not match the expected peg-gemma4 format", raise max_tokens rather than assuming the quant is broken.

All builds compared

Same model, same prompts, same machine (Apple M5 Max, 128 GB). Generation tok/s; quality is mean NLL on held-out Bokmål vs an unquantized bf16 MLX reference (lower is better).

_{* GGUF vision needs the separate mmproj file from the GGUF repo.}

On Apple Silicon MLX is meaningfully faster than GGUF at the same precision — 94 vs 75 tok/s at 8-bit, on identical hardware and prompts. MLX also won the Nynorsk OCR comparison: it transcribed gjere correctly where GGUF Q8_0 produced bokmål gjøre.

GGUF is the right choice if you are not on Apple Silicon, or if you want the smallest build — Q4_K_M at 16.8 GB has no MLX equivalent, because uniform 4-bit MLX quantization breaks this model (NLL 2.47 → 9.50) while llama.cpp's K-quants protect the attention and dense-MLP tensors that are the sensitive ones here.

Credit

Borealis 2 was built by the AI Lab at the National Library of Norway (Nasjonalbiblioteket), from google/gemma-4-26B-A4B-it, continued pre-trained on Aurora and SFT'd on NbAiLab/aurora-sft-2606.

All credit belongs to them — https://ai.nb.no · original repo · [email protected]. I only ran the format conversion.

Conversion details

mlx_vlm convert --hf-path <local copy of NbAiLab/borealis2-26b-a4b-preview> \
  -q --q-bits 8 --mlx-path borealis2-26b-a4b-preview-MLX-8bit-vision

• mlx-vlm 0.6.15, mlx 0.32.1, transformers 5.15.1
• The upstream repo ships no preprocessor_config.json; mlx-vlm supplies its own gemma4 processor, so nothing about image preprocessing was guessed. (This vision tower standardizes internally — vision_config.standardize: true with std_bias/std_scale in the weights.)

Other builds: MLX text-only · GGUF for llama.cpp