karume-gemma4

Gemma 4 Text Decoder — Karume

What is this

A chat distribution: the text decoder of google/gemma-4-E2B-it, converted into the WebGPU inference runtime Karume's container format (a .krm part sequence whose first part carries the graph and model descriptors). Runs as-is in the browser and in Deno — string in, string out.

• Tokenizer and chat format ship with the weights. chat() renders the turns, encodes them, samples, and decodes incrementally, so callers hand over messages and read back text fragments as they are decided.
• Per-layer embeddings are container assets, not graph weights. They are gathered on the host and handed to the graph as an input, which keeps them out of the GPU's resident set; the loader reads only the vocabulary ranges a conversation actually touches.
• Context: the default capacity is 4096 tokens per conversation, and the loader accepts any capacity between the chunk length in use and 131072 (the model's declared position limit); rotary cos/sin are generated on the host per chunk from the declared rope parameters, so no position table ships. Prefill runs in chunks of 768 rows by default, and the chunk length itself can be raised up to 768 — the traced upper bound of the chunk symbol, which the graph does not carry.
• Not multimodal here. The published checkpoint also carries vision and audio towers; this distribution contains neither, and cannot take images or audio.
• Not readable by transformers (it's a different container with an embedded graph); the reader is a pipeline that implements gemma4/1.
• Exporter used for the conversion: karume/0.13.0. The distribution manifest is karume.json (karume/5).

Base weights and attribution

Converted and quantized from the upstream checkpoint — the original weights are not distributed here.

• e2b: google/gemma-4-E2B-it, licensed Apache 2.0 (license / full text; a verbatim copy is in LICENSE.md).
• assistant: google/gemma-4-E2B-it-assistant, the multi-token-prediction drafter head, licensed Apache 2.0 under the same terms. It is redistributed here with its clustered sparse output head replaced by a dense projection over the full vocabulary, and it reads the key/value states and embedding table of the main model rather than carrying its own. Its own weights are quantized to int8 throughout, linear layers included, rather than to packed int4.
• Changes made here (also listed in NOTICE.md, per Apache 2.0 §4(b)): the text decoder was extracted and re-expressed in the Karume container format; linear weights were quantized to packed int4 (group 32) and the embedding tables to int8; the per-layer embeddings were moved out of the graph into container assets that the host gathers; the exit was narrowed to the selected rows' logits and hidden states; rotary cos/sin arrive as host-generated inputs. No retraining and no fine-tuning.
• What the model can and cannot do — capabilities, limitations, intended use, evaluations and safety guidance — is documented on the upstream model card, and this repository does not restate it: read google/gemma-4-E2B-it before using it.

Models

model selects one of these; omitted, it is e2b. quant defaults to that model's own default quant.

Usage

import { Gemma4Pipeline } from "jsr:@karume/models/gemma";

await using pipeline = await Gemma4Pipeline.fromPretrained({
  repo: "hdae/karume-gemma4",
  // Pin a commit for reproducible builds — without it you track `main`, and a future
  // repo update (renamed files, new manifest format) may break your app.
  // Copy the full hash from this repo's "Files and versions" tab:
  // revision: "<full commit sha>",
}, {
  // model: "e2b", // default — available: e2b
  // quant: "i4-fast", // default — available: i4 / i4-fast / i4-gemvpar
});

const stream = pipeline.chat([
  { role: "user", content: "What is the capital of France?" },
], {
  maxNewTokens: 128,
  // sampler: { temperature: 0 }, // greedy — overrides the recommended default below
});

// Fragments arrive as the decoder settles them (multi-byte characters are held back
// until they are complete).
for await (const text of stream) Deno.stdout.write(new TextEncoder().encode(text));
console.log(await stream.done); // { reason: "eos" | "max-tokens" | "aborted", … }

Messages are plain system / developer / user / assistant turns; tool calls, thinking channels and image or audio parts are rejected rather than silently dropped. With no sampler in the request, generation uses this repository's recommended default: temperature 1.0, top-k 64, top-p 0.95. Weights are fetched once and cached (verified against karume.json's size / sha256).

Model: e2b

Quants

If no quant is given, it runs as i4-fast (this model's recommended default). Per-file size and sha256 live in karume.json — verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own.

Generation

Derived from the exported graph and the checkpoint's own generation_config.json, and checked against each other when this repository was assembled.

• context: 4096 tokens per conversation (prompt + generated) by default; any capacity from the chunk length in use up to 131072 can be chosen at load time
• prefill chunk: 768 rows per step by default, up to 768 (the traced upper bound of the chunk symbol)
• recommended sampler: temperature 1.0, top-k 64, top-p 0.95 — used when a request omits sampler; pass { temperature: 0 } for greedy decoding
• stop tokens come from the tokenizer asset in this repository, not from the caller
• device limits: maxBufferSize ≥ 402,653,184 B / maxStorageBufferBindingSize ≥ 402,653,184 B (the largest single tensor must bind)