Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC

Pelnora/Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC

MLC-LLM q4f16_1 conversion of speakleash/Bielik-Minitron-7B-v3.0-Instruct for in-browser inference via WebLLM (WebGPU) and local Apple-Silicon Metal inference via mlc_llm. Published as a Pelnora brand asset — a pure inference-format port: no fine-tuning, no additional training data.

🇵🇱 Konwersja modelu speakleash/Bielik-Minitron-7B-v3.0-Instruct do formatu MLC q4f16_1, żeby dało się go uruchomić w przeglądarce na WebGPU (przez WebLLM) oraz lokalnie na Metalu (Apple Silicon). To wyłącznie zmiana formatu pod inferencję — bez dotrenowania, bez nowych danych.

Why this model

Polish legal Q&A baseline for in-browser inference. When you need a model that runs entirely client-side (WebGPU, no server round-trip, no data leaving the browser) and actually knows Polish law, this is it. Concrete evidence: smoke test #6 (explain art. 101² Kodeksu pracy), head-to-head vs Gemma 4 E4B-it (transformers.js + WebGPU) on the identical prompt —

• Bielik-Minitron-7B correctly identifies art. 101² as the post-employment non-compete clause (employer must inform the employee in advance; breach has consequences; it is set by agreement) — and does not confuse it with art. 101¹ (the in-employment duty).
• Gemma 4 E4B-it returned "please paste the text of the article" — i.e. no knowledge of the Polish labour code at all (a 140-language model vs. Bielik's ≈292B Polish-token diet; this is a training-data gap, not a runtime bug).

Reference: pelnora-gemma4-spike/DECISION_REPORT.md (the team's Gemma 4 spike report) and quality_minitron_smoke_results.txt in the conversion repo. Net: Bielik MLC stays the answer for Polish legal work; Gemma 4's only distinct edge — multimodal OCR of scans — is a separate scope.

What this is

• Source model: speakleash/Bielik-Minitron-7B-v3.0-Instruct — a SpeakLeash / Cyfronet model produced by NVIDIA-Minitron-style pruning + knowledge distillation from Bielik-11B-v3-Base (which itself uses the Mistral-7B-v0.2 architecture with depth up-scaling). It is Llama-architecture compatible (LlamaForCausalLM, attention_bias=false, mlp_bias=false, 40 layers, hidden 4096, intermediate 11264, GQA 32 query / 8 KV heads, head_dim 128, vocab 32128, rope_theta 1e6, 32 768 context, ChatML chat template).
• This repo: the same weights quantized to q4f16_1 (4-bit grouped weights, fp16 scales) with mlc_llm convert_weight, an mlc-chat-config.json (chatml_nosystem conversation template, stop tokens patched — see below), and the precompiled model libraries for WebGPU (.wasm) and Metal (.dylib).
• Footprint: ~3.9 GB of weight shards (params_shard_*.bin × 124), ≈ 4.0 GB GPU memory for the parameters, ≈ 5.0 GB without a KV cache / ≈ 5.7 GB with a 4 K KV cache. Fits comfortably on a 16 GB Apple-Silicon Mac and on a WebGPU adapter with a ~4 GB maxBufferSize.

Files

mlc-chat-config.json                                   # MLC chat config (conv template, stop tokens, sampling defaults)
tokenizer.json, tokenizer.model, tokenizer_config.json, added_tokens.json
tensor-cache.json                                      # param manifest (MLC's ndarray-cache equivalent)
params_shard_0.bin … params_shard_123.bin             # ~3.9 GB q4f16_1 weights
Bielik-Minitron-7B-v3-Instruct-q4f16_1-webgpu.wasm    # WebGPU model library (~6.8 MB) — for WebLLM
Bielik-Minitron-7B-v3-Instruct-q4f16_1-metal.dylib    # Apple-Silicon Metal model library (~3.4 MB) — for local mlc_llm

⚠️ stop_token_ids patch (read this if you regenerate the config)

This vocab puts <|im_end|> — the real ChatML turn boundary — at token id 32001, not at id 2 (which here is </s>). mlc_llm gen_config's chatml_nosystem template defaults stop_token_ids to [2]; left as-is, the model never stops at <|im_end|> and runs all the way to max_tokens on every turn. mlc-chat-config.json in this repo is patched to stop_token_ids: [32001, 2] (matching config.json's eos_token_id) plus stop_str: ["<|im_end|>"]. If you re-run gen_config against the source model, re-apply that patch — it is a non-obvious landmine. (The earlier Bielik-4.5B-v3 MLC build hit the same class of bug with id 4.)

Browser usage (WebLLM)

The model library and weights aren't in binary-mlc-llm-libs / mlc-models, so register them explicitly via appConfig (model = this HF repo; model_lib = the .wasm in it):

import * as webllm from "@mlc-ai/web-llm";

const MODEL_ID = "Pelnora/Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC";
const REPO = "https://huggingface.co/Pelnora/Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC";

const appConfig = {
  model_list: [
    {
      model: REPO,                       // weights + mlc-chat-config.json + tokenizer
      model_id: MODEL_ID,
      model_lib: `${REPO}/resolve/main/Bielik-Minitron-7B-v3-Instruct-q4f16_1-webgpu.wasm`,
      // first-visit download is the ~3.9 GB of params_shard_*.bin; cached in the browser afterwards
    },
  ],
};

const engine = await webllm.CreateMLCEngine(MODEL_ID, { appConfig });
const reply = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "Jesteś pomocnym asystentem prawnym. Odpowiadasz po polsku." },
    { role: "user", content: "Wyjaśnij art. 101² Kodeksu pracy w prostym języku, maksymalnie 5 zdań." },
  ],
  temperature: 0.6,
  top_p: 0.9,
});
console.log(reply.choices[0].message.content);

Self-hosting the files instead of HF: WebLLM resolves the config at <model>/resolve/main/mlc-chat-config.json and expects the param manifest as ndarray-cache.json (this repo ships it as tensor-cache.json — symlink/rename if your static host needs the legacy name), and a cross-origin-isolated context (COOP: same-origin + COEP: require-corp) for WebGPU + WASM threads.

Local usage (Apple Silicon, Metal)

from mlc_llm import MLCEngine

engine = MLCEngine(
    model="Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC",
    model_lib="Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC/Bielik-Minitron-7B-v3-Instruct-q4f16_1-metal.dylib",
)
for chunk in engine.chat.completions.create(
    messages=[{"role": "user", "content": "Napisz haiku o jesieni w Warszawie."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)
engine.terminate()

Conversion details

Smoke tests — quality, and head-to-head vs Gemma 4 E4B

8 prompts (3 generic + 5 Polish-legal — the same suite the team ran against Gemma 4 E4B), via mlc_llm's MLCEngine on the Metal lib, temperature=0.6 / top_p=0.9, all in one engine session. Engine load ≈ 7 s on the M5. Rating: ✓ good / ~ partial / ✗ bad. Full transcripts: quality_minitron_smoke_results.txt in the conversion repo.

8/8 functionally PASS (coherent, on-task, correct Polish; clean stop on <|im_end|> in every prompt not capped by max_tokens); ~15–26 tok/s on the M5 Metal build. (#6's minor caveat: the art. 101² §3 mechanics — the employer owes the employee a ≥25% indemnity for the restraint — are stated a bit loosely; the gist is right.)

vs Gemma 4 E4B (transformers.js + WebGPU; team spike, 2026-05-10): comparable on general Polish and legal-text tasks (#1–5), but ahead on legal knowledge (#6 above) and ahead on simply finishing the run — Gemma's ONNX/transformers.js session self-disposed after ~3–4 generations (#7 and #8 were never reached) while Bielik ran all 8 in one session. Bielik MLC is also smaller on first-visit download (~3.9 GB vs ~4.85 GB) at comparable decode speed. Gemma 4's one distinct advantage — multimodal OCR of scanned documents — is outside this comparison.

(For reference vs the earlier Bielik-4.5B-v3 qwen2-fallback conversion: this 7B build is the clean-architecture port — --model-type llama, 0 dropped biases vs the 4.5B's 240 — and it shows: #5 clause classification passes zero-shot here where the 4.5B failed it, and #7 JSON comes back without the spurious ```json wrapper the 4.5B always emitted.)

License & attribution

Licensed under Apache License 2.0, inheriting from the source model speakleash/Bielik-Minitron-7B-v3.0-Instruct. Redistribution as a Pelnora brand asset; quantization and packaging only — no model weights were retrained.

NOTICE

This work builds on, and preserves attribution to:

• SpeakLeash (Spichlerz) & ACK Cyfronet AGH — Bielik-Minitron-7B-v3.0-Instruct and the Bielik-11B-v3 family it is distilled from. Bielik models were trained on a large Polish-language corpus by the SpeakLeash community on the Cyfronet PLGrid infrastructure. Please also credit them per the source model card if you use this conversion.
• NVIDIA — the Minitron methodology (structured pruning + knowledge distillation of large language models) that produced the 7B model from the 11B base.
• Mistral AI — the Mistral-7B-v0.2 architecture underlying Bielik-11B-v3-Base (depth-up-scaled).
• MLC-AI team — mlc-llm / web-llm / Apache TVM, used for the q4f16_1 quantization and the WebGPU / Metal model libraries.

If you cite Bielik, use the citation block from the upstream speakleash/Bielik-Minitron-7B-v3.0-Instruct model card.