Bharat-Tiny-LLM-v3-ONNX

Bharat-Tiny-LLM v3 ONNX (WebGPU)

ONNX weights for eulogik/Bharat-Tiny-LLM-v3, a 1.7B bilingual Hindi/Hinglish/English edge LLM (Qwen3-1.7B-Base + 326 Devanagari tokens, total vocab 151,969), converted for in-browser inference with Transformers.js on WebGPU.

Recipe

1. FP32 ONNX export of the merged model via optimum-cli export onnx (text-generation-with-past), validated against PyTorch logits.
2. 4-bit symmetric weight quantization, block size 32 (MatMulNBitsQuantizer), same settings as onnx-community conversions.
3. Embedding table rounded to FP16 (1.24 GB -> 0.62 GB).
4. Verified: onnx.checker passes and ONNX Runtime loads the model (59 inputs / 57 outputs: input_ids, attention_mask, past key/values).

Layout follows the onnx-community convention (onnx/model_q4f16.onnx).

Usage

// npm i @huggingface/transformers
import { pipeline } from '@huggingface/transformers';

const generator = await pipeline(
  'text-generation',
  'eulogik/Bharat-Tiny-LLM-v3-ONNX',
  { device: 'webgpu', dtype: 'q4f16' },
);

const out = await generator(
  [{ role: 'user', content: 'नमस्ते, आप कैसे हैं?' }],
  { max_new_tokens: 200, temperature: 0.4, top_p: 0.9 },
);
console.log(out[0].generated_text.at(-1).content);

Live demo: eulogik/bharat-tiny-llm-v3-demo

License: Apache-2.0.