borealis2-26b-a4b-preview-GGUF

borealis2-26b-a4b-preview-GGUF

GGUF quantizations of NbAiLab/borealis2-26b-a4b-preview, the Norwegian-centric preview model from the AI Lab at the National Library of Norway (Nasjonalbiblioteket).

This is a format conversion, not a new model. Nothing was retrained, fine-tuned or merged. All behaviour, quality and limitations come from the original NbAiLab release.

Unlike the MLX builds, these run anywhere llama.cpp runs — Linux, Windows, macOS, CPU or GPU.

Files

Plus mmproj-borealis2-26b-a4b-preview-BF16.gguf (1.19 GB) — the optional vision projector, see Vision / OCR below. You do not need it for text-only use.

Speeds are generation tok/s measured on an Apple M5 Max (128 GB) with Metal, llama-server -ngl 99 -fa on -c 8192, sampling at temp 1.0 / top-k 64 / top-p 0.95. Your numbers will differ.

Norwegian-calibrated importance matrix

The K-quants here (Q6_K, Q4_K_M) were built with an importance matrix calibrated on Norwegian text, not the usual English-only calibration. This model is Norwegian-specialized, so an English imatrix would weight the wrong activations.

The calibration corpus was 2.04 MB of mixed text — 45% Bokmål, 35% Nynorsk, 20% English — drawn from Norwegian Bokmål Wikipedia, Nynorsk Wikipedia and English WikiText-2, with documents shuffled so each imatrix chunk sees a mix of languages. It was computed over 1194 chunks at n_ctx=512 against the BF16 model.

Q8_0 does not use an importance matrix — it doesn't need one.

Running it

LM Studio

Search for BobTheShoplifter/borealis2-26b-a4b-preview-GGUF in the model browser, or drop a .gguf into ~/.lmstudio/models/BobTheShoplifter/borealis2-26b-a4b-preview-GGUF/. Verified loading in LM Studio's llama.cpp runtime.

llama.cpp

llama-server -m borealis2-26b-a4b-preview-Q8_0.gguf --jinja -ngl 99 -fa on -c 8192 \
  --temp 1.0 --top-k 64 --top-p 0.95

The --jinja flag matters: it makes llama.cpp use the embedded Gemma 4 chat template rather than a built-in guess.

Sampling settings

temperature = 1.0    top_k = 64    top_p = 0.95

From the model's own generation_config.json. In testing, plain temperature sampling without top_k/top_p produced repetition loops on longer generations. If you see looping, check your sampler first.

Chat template and stop tokens

The upstream repo keeps its chat template in a separate chat_template.jinja (transformers 5.x style) rather than inside tokenizer_config.json. It converted correctly — convert_hf_to_gguf.py reads that file and embeds it as tokenizer.chat_template in every GGUF here (18681 characters, matching the source). No gguf_set_metadata fix or --chat-template-file workaround was needed. Just remember --jinja.

Gemma 4 uses <|turn> … <turn|> turn markers and a <|channel>thought … <channel|> reasoning channel, not Gemma 2/3's <start_of_turn>. llama.cpp recognises <eos>, <turn|> and <|tool_response> as end-of-generation tokens for the gemma4 architecture, so generation stops cleanly without extra configuration.

Note for the MLX builds: LM Studio's MLX runtime reads eos_token from tokenizer_config.json and would leave a trailing <turn|> visible. That is fixed in the MLX repos by declaring <turn|> as eos_token. The GGUF path never had this problem.

Reasoning channel

On harder prompts the model often writes a reasoning block — frequently in English — before its answer, even though the template's default enable_thinking=false tries to suppress it. The real answer follows the closing <channel|>. This is upstream behaviour, not a quantization artefact.

Vision / OCR (optional)

The upstream repo ships a gemma4_vision tower but no preprocessor_config.json, and the instruction tuning was text-only — so it is easy to assume vision is unusable here. It isn't.

mmproj-borealis2-26b-a4b-preview-BF16.gguf (1.19 GB) in this repo is the vision projector, built from this model's own vision weights. Pair it with any of the GGUFs above:

llama-server -m borealis2-26b-a4b-preview-Q8_0.gguf   --mmproj mmproj-borealis2-26b-a4b-preview-BF16.gguf   --jinja -ngl 99 -fa on -c 8192

The .gguf weights work fine without the mmproj if you only want text — it is a separate, optional download.

Two things worth knowing:

• No preprocessing parameters were guessed. This vision tower standardizes internally (the vision_config sets standardize: true and the weights carry std_bias / std_scale), so image mean/std are the identity [0,0,0] / [1,1,1]. That matches Google's own published mmproj for gemma-4-26B-A4B-it byte for byte on every clip.* metadata key.
• The vision tower is unchanged from the Gemma 4 base. NbAiLab's continued pre-training and SFT were text-only, and the vision tensors are bit-identical to the base model's. So image understanding is stock Gemma 4 — what is new is that the language side answering about the image is Norwegian-specialized.

Tested on rendered Nynorsk newspaper text: transcription was accurate including æ ø å, and it summarized the content in correct Nynorsk. Bear in mind the SFT never included image-text pairs, so treat vision as a capability inherited from the base rather than one NbAiLab tuned for.

⚠️ Preview quality and safety — please read

Carried forward from the original model card, and not boilerplate:

• This is a preview experiment, not a production model. Outputs may be unstable and may hallucinate.
• It has not been fully safety-aligned. It may produce harmful, biased or offensive content, and may follow harmful instructions.
• Do not use it for safety-critical or high-stakes applications. Add your own safety mitigations before deploying anything on top of it.
• It is an early SFT checkpoint, not a final release.

In testing, on open-ended "write a long article" prompts the model is unreliable across random seeds — it sometimes stops after only a title, sometimes emits degenerate text. This was verified against unquantized BF16 and the behaviour is identical, so it belongs to the preview checkpoint rather than to these quantizations. Short and medium-length instructions behaved well.

License — read before redistributing

This model is not Apache 2.0. It is released under the NB-License 1.0, an adaptation of Apache 2.0 with additional use-based restrictions. On top of the usual Apache terms, you must not:

• intentionally use the model to recreate data it was trained on; or
• use the model or its output to power end-user services whose primary purpose is giving access to licensed press publications contained in the training data.

Redistribution of derivatives is permitted, but the license travels with the model. The full LICENSE and LICENSE_FAQ.pdf are included here, copied verbatim from the source repository.

Two gotchas worth knowing

llama-server can return HTTP 500 on truncated output. llama.cpp parses this architecture's replies with a strict peg-gemma4 grammar. If a generation is cut off mid-structure (e.g. you set max_tokens low enough that the model is still inside its reasoning channel), the server rejects the whole response with "The model produced output that does not match the expected peg-gemma4 format". Give it enough tokens to finish, or use the raw /completion endpoint. LM Studio uses its own reasoning parser and is not affected.

MoE expert coverage in the imatrix. With 128 experts and top-8 routing, a handful of experts never activate during calibration. Across layers, coverage was 96–99% (1–5 of 128 experts per layer had no data); llama.cpp falls back to default quantization for those. This is normal for wide MoE models and the quants test clean.

Reasoning — read this before choosing a build

This model can reason, but it is off by default, and which runtime you use decides whether you can turn it on at all.

Upstream's chat_template.jinja sets enable_thinking | default(false), which pre-closes the thought channel with <|channel>thought\n<channel|> so the model answers immediately. Setting enable_thinking=true injects <|turn>system\n<|think|>\n<turn|> and the model reasons first.

What actually works — measured, same prompt and settings on one machine

LM Studio + MLX cannot reason

llama.cpp implements Gemma 4's reasoning channel at the architecture level — the same code that registers <eos>, <turn|> and <|tool_response|> as end-of-generation for gemma4. LM Studio's llama.cpp runtime inherits that and folds reasoning into a collapsible block automatically.

LM Studio's MLX runtime has no equivalent, so it never enables thinking and never parses the channel. Identical prompt and sampling, same machine:

This affects every gemma4 MLX model in LM Studio, not just this one. It is a runtime gap, not something a quantization or a model.yaml can fix — setting enable_thinking in the template only makes the raw <channel|> markers leak into the reply, because nothing parses them.

If you want reasoning — for agents especially — use the GGUF builds. The MLX builds remain the better choice for everything else on Apple Silicon: faster (94–110 vs 75–109 tok/s) and more accurate on Norwegian OCR.

# recommended for agents
llama-server -m borealis2-26b-a4b-preview-Q6_K.gguf \
  --jinja --reasoning on -ngl 99 -fa on -c 8192 \
  --temp 1.0 --top-k 64 --top-p 0.95

Budget tokens generously — the reasoning block is often longer than the answer, and a truncated one leaves content empty. The model tends to reason in English even when answering in Norwegian; that is upstream behaviour.

llama-server parses replies with a strict peg-gemma4 grammar and returns HTTP 500 if a generation is cut off mid-structure. If you see "output that does not match the expected peg-gemma4 format", raise max_tokens rather than assuming the quant is broken.

All builds compared

Same model, same prompts, same machine (Apple M5 Max, 128 GB). Generation tok/s; quality is mean NLL on held-out Bokmål vs an unquantized bf16 MLX reference (lower is better).

_{* GGUF vision needs the separate mmproj file from the GGUF repo.}

On Apple Silicon MLX is meaningfully faster than GGUF at the same precision — 94 vs 75 tok/s at 8-bit, on identical hardware and prompts. MLX also won the Nynorsk OCR comparison: it transcribed gjere correctly where GGUF Q8_0 produced bokmål gjøre.

GGUF is the right choice if you are not on Apple Silicon, or if you want the smallest build — Q4_K_M at 16.8 GB has no MLX equivalent, because uniform 4-bit MLX quantization breaks this model (NLL 2.47 → 9.50) while llama.cpp's K-quants protect the attention and dense-MLP tensors that are the sensitive ones here.

Credit

Borealis 2 was built by the AI Lab at the National Library of Norway (Nasjonalbiblioteket), starting from google/gemma-4-26B-A4B-it, continued pre-trained on the Aurora corpus and supervised fine-tuned on NbAiLab/aurora-sft-2606.

All credit for the model belongs to them — see https://ai.nb.no and the original repository. Contact: [email protected].

I only ran the format conversion.

Build details

• llama.cpp b1-353b32d (2026-08-21), built with -DGGML_METAL=ON
• convert_hf_to_gguf.py → BF16 GGUF → llama-quantize
• Requires transformers >= 5.0 to run the converter; llama.cpp's pinned transformers==4.57.6 cannot read this repo's tokenizer (extra_special_tokens is a list in the transformers-5 format).

Also available: MLX builds for Apple Silicon.