MiniCPM5-1B-MLC

MiniCPM5-1B-MLC

openbmb/MiniCPM5-1B converted to MLC format so it runs in the browser with WebLLM on WebGPU. Weights, model libraries (.wasm) and the chat/tokenizer config are all in this repo, so WebLLM can load everything straight from here.

Built with mlc-llm for web-llm 0.2.85 (model library ABI v0_2_84).

Variants

• q4f16_am — asym (K-quant-like) int4 + asym int6 qkv/down on use_more_bits layers + asym int5 lm_head
• q4f32_am — asym (K-quant-like) int4 + asym int6 qkv/down on use_more_bits layers + asym int5 lm_head [fp32 activations]

Use with WebLLM

import * as webllm from "@mlc-ai/web-llm";

const appConfig: webllm.AppConfig = {
  model_list: [
    {
      "model": "https://huggingface.co/CharlZKP/MiniCPM5-1B-MLC/resolve/main/MiniCPM5-1B-q4f16_am-MLC",
      "model_id": "MiniCPM5-1B-q4f16_am-MLC",
      "model_lib": "https://huggingface.co/CharlZKP/MiniCPM5-1B-MLC/resolve/main/libs/MiniCPM5-1B-q4f16_am_cs1k-webgpu.wasm",
      "vram_required_MB": 884.27,
      "low_resource_required": true,
      "overrides": {
        "context_window_size": 4096
      },
      "required_features": [
        "shader-f16"
      ]
    },
    {
      "model": "https://huggingface.co/CharlZKP/MiniCPM5-1B-MLC/resolve/main/MiniCPM5-1B-q4f32_am-MLC",
      "model_id": "MiniCPM5-1B-q4f32_am-MLC",
      "model_lib": "https://huggingface.co/CharlZKP/MiniCPM5-1B-MLC/resolve/main/libs/MiniCPM5-1B-q4f32_am_cs1k-webgpu.wasm",
      "vram_required_MB": 984.94,
      "low_resource_required": true,
      "overrides": {
        "context_window_size": 4096
      }
    }
  ]
};

const engine = await webllm.CreateMLCEngine("MiniCPM5-1B-q4f16_am-MLC", { appConfig });
const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(reply.choices[0].message.content);

The same records are in webllm-model-records.json. model points at the folder of one quantization and model_lib at its .wasm; nothing else has to be hosted.

Files

MiniCPM5-1B-q4f16_am-MLC/  weights (params_shard_*.bin), tensor-cache.json, mlc-chat-config.json, tokenizer
MiniCPM5-1B-q4f32_am-MLC/  weights (params_shard_*.bin), tensor-cache.json, mlc-chat-config.json, tokenizer
libs/                      model libraries (.wasm), one per quantization
webllm-model-records.json  ready-to-paste WebLLM ModelRecords
QUALITY.md                 quantization quality measurements

About the model

MiniCPM5-1B is a 1.08B-parameter Llama-architecture model with a 130k vocabulary, a native 131k context and a hybrid thinking / no-thinking chat mode. See the original model card for what it can do.

Quality

q4f16_am is an asymmetric K-quant-like format (per 32 weights an 8-bit scale and an 8-bit minimum, per 256 weights a bf16 pair), with more bits on the tensors that need them, and scales chosen with an importance matrix from calibration text. It is the same size as GGUF Q4_K_M and closer to the original model on every measurement we made.

KLD is the KL divergence of the next-token distribution against the original bf16 model, so lower is better. "Same token" is how often greedy decoding picks the same next token as bf16.

Measured on wikitext prose and Python code with the same quantizer the conversion uses. Method, full table and caveats are in QUALITY.md. Task-level benchmarks such as MMLU or GSM8K were not run.

Both variants hold the same quantized weights. q4f32_am computes activations in fp32 for GPUs without the WebGPU shader-f16 feature, so it is slower but runs anywhere.

Thinking mode

By default the model reasons first and emits <think>…</think>. To turn that off per request:

const reply = await engine.chat.completions.create({
  messages,
  extra_body: { enable_thinking: false },   // same as HF enable_thinking=False
});

WebLLM keeps earlier reasoning in the chat history, so strip it before the next turn:

const stripThinking = (messages) =>
  messages.map((m) => m.role === "assistant"
    ? { ...m, content: m.content.replace(/<think>[\s\S]*?<\/think>\s*/g, "") }
    : m);

Upstream suggests temperature 0.9 and top_p 0.95 when thinking, and 0.7 / 0.95 when not. The config here defaults to 0.7 / 0.95.

Context and memory

• The default context is 4096 tokens. Raise it per app with overrides.context_window_size, up to the native 131072. The compiled model library does not depend on it.
• The KV cache costs about 24.6 KB per token in f16, so 4k is roughly 100 MB and 32k about 800 MB.
• About 0.9 GB of VRAM at 4k context for q4f16_am.
• The largest single GPU buffer is 95.6 MiB, which fits devices whose WebGPU maxStorageBufferBindingSize is 128 MiB, such as many phones.

Chat format

ChatML with a leading <s>, generation stops at <|im_end|> or </s>. WebLLM always renders a system block, so the config carries a short default system prompt: "You are a helpful assistant."

Validation

• Runs in real WebLLM 0.2.85: q4f16_am on an f16-capable WebGPU adapter, q4f32_am in headless Chromium, both answering correctly.
• q4f32_am produces exactly the same first 24 greedy tokens as PyTorch running the same stored weights.
• All 103 WGSL kernels of the f16 model library compile to WebGPU pipelines, and both libraries pass an import/export ABI check against an official WebLLM library.
• Not yet benchmarked for speed on a discrete GPU.

How this was built

mlc_llm convert_weight / gen_config / compile (WebGPU) from a conversion pipeline with custom quantization presets, using an importance matrix from calibration text. The .wasm libraries contain the model's compiled kernels, including the custom dequantization kernel, and are checked against an official WebLLM library for ABI compatibility.

License

apache-2.0, inherited from openbmb/MiniCPM5-1B.