kev-0.5b-onnx

kev-0.5b-web — browser-compatible decision model

The kev decision model (Qwen2.5-0.5B backbone + LoRA + pointer readout, prefill-only — it never generates text) exported to a single fp16 ONNX graph that runs in the browser via onnxruntime-web (WebGPU or WASM). Built for the Chrome MV3 extension in this repo's browser/ directory; consumable by any JS/WASM client.

What this repo contains

Graph interface

input_ids        int64   [B, L]
position_ids     int64   [B, L]      branch positions restart after the shared state prefix
attention_mask   fp16    [B, 1, L, L]  CUSTOM additive block-causal mask (see below)
->
last_hidden_state fp16   [B, L, 896]

attention_mask is not the usual 2D padding mask. It encodes the block-causal structure: a token at segment k (a question branch) attends to the shared state prefix (segment 0) and to its own segment only; the state prefix attends causally to itself. Allowed = 0x0000, masked = 0xFBFF (−65504, the fp16 min finite). The client builds it from the packed record; see browser/engine.js (branchMask).

How decisions are read out (client-side)

The graph returns hidden states only. The pointer head is applied in JS:

logits[option] = (k(h_opt) · q(h_decide)) * scale     # q,k: 896 -> 256
probs = softmax(logits)

at the decide token and each option's closing optend token of every question branch. One forward pass answers all questions packed against a shared state (the "system one" primitive).

Usage (transformers.js + the extension engine)

import { AutoTokenizer } from "@huggingface/transformers";
import * as ort from "onnxruntime-web";

const REPO = "https://huggingface.co/<user>/kev-0.5b-web/resolve/main/";
const tok  = await AutoTokenizer.from_pretrained("<user>/kev-0.5b-web");
const head = await (await fetch(REPO + "head.json")).then(r => r.json());
const meta = await (await fetch(REPO + "meta.json")).then(r => r.json());
const sess = await ort.InferenceSession.create(REPO + "onnx/model_fp16.onnx",
  { executionProviders: ["webgpu", "wasm"] });
// then: pack record -> encode/branchMask -> sess.run -> readout  (see browser/engine.js)

The extension in browser/extension/ loads these same files from its local model/ dir; point it at this repo to avoid shipping ~1 GB inside the zip.

Precision / fidelity

• Exported in fp16 (native for WebGPU; fits the browser WASM heap — the fp32 graph is ~2 GB and OOMs a Chrome renderer).
• fp16 vs the PyTorch checkpoint: max abs probability error ~5e-4; decisions identical. 2-decimal probabilities can differ by ±0.01.
• Verified in real Chrome (WASM backend): all reference decisions match the checkpoint.

Backend notes

• WebGPU: ~18× faster than WASM CPU on real GPU hardware.
• WASM CPU: portable fallback, correct and stable everywhere (~3 s prefill for 60 tokens on CPU).
• Under software rasterizers (SwiftShader, no GPU) the WebGPU EP can return zeroed output after the first run — prefer WASM there.

Provenance

• Base: Qwen/Qwen2.5-0.5B (Apache-2.0).
• Adapter + pointer head: jaredpalmer/kev-0.5b (Apache-2.0), merged into the exported graph.
• Export: browser/export/export_fp16.py (torch.onnx.export, opset 17, dynamo=False).
• This is a prototype decision model, not a chat/assistant. No text generation. See the parent MODEL_CARD.md for training data, evaluation, and limitations.

License

Apache-2.0 (inherited from the base model and the kev checkpoint).