laya-multilingual-onnx-int8

Laya Multilingual — INT8 ONNX for CPU (untested)

This repository contains a smaller INT8 ONNX version of convaiinnovations/laya-multilingual. It is designed for JavaScript and TypeScript programs running on a CPU with Node.js.

Decision-making demo: Laya Tetris Lab

Sharjeelbaig/laya-tetris-ai is a small local web demo that uses this model to choose among legal Tetris placements. The game simulates the physics and supplies measurable outcomes—cleared lines, holes, height, and surface bumpiness—then Laya selects the final plan from a short list.

The demo is exploratory rather than a benchmark: this INT8 model is still untested against the FP32 reference. See the Tetris repository for local setup and source code.

Important: untested model

This file was created by automatic dynamic INT8 quantization of the verified FP32 ONNX model. Its file structure was checked, and the TypeScript examples compile, but its predictions have not yet been tested or compared with the original model. Do not use it for production, safety decisions, or customer-facing actions until you test it on your own representative examples and compare it with the FP32 model.

The original FP32 export is available at Sharjeelbaig/laya-multilingual-onnx. Use that as the reference when evaluating this INT8 version.

Why use INT8?

The original model is like storing every number with four bytes (FP32). INT8 stores many weights with one byte plus small scaling information. That makes this file about 317 MB instead of about 1.2 GB. The trade-off is that it may change predictions or confidence values slightly—hence the testing warning above.

Install and run

git clone https://huggingface.co/Sharjeelbaig/laya-multilingual-onnx-int8
cd laya-multilingual-onnx-int8
npm install
npm run example

The runtime uses onnxruntime-node, which means it runs in Node.js (server, local CLI, or Electron). It is not intended for a normal web page browser.

General routing example

import { LayaOnnx } from './src/index.js';

const model = await LayaOnnx.fromDirectory('/models/laya-multilingual-onnx-int8');
const result = await model.predict(
  { body: 'I was charged twice for invoice 4411. Please refund me.' },
  {
    type: 'choice',
    instructions: 'Which team should handle body?',
    criteria: {
      billing: 'invoices, payments, and refunds',
      technical: 'bugs and outages',
      sales: 'pricing and purchase questions',
    },
  },
);

console.log(result.choice);
console.log(result.probabilities);

AI-agent safety gate example

Run the included example:

npm run safety-gate

It demonstrates how an agent can route email, refund, and deletion actions to allow, review, or block. It is an example project, not a complete security system. Keep deterministic permission checks, audit logs, rate limits, and human review. Because this INT8 export is untested, configure the example to send every decision to human review until evaluation is complete.

Included files

• model.onnx — dynamically quantized INT8 ONNX model for CPU inference.
• tokenizer/ — original multilingual tokenizer.
• src/index.ts — Node.js TypeScript runtime wrapper.
• examples/basic.ts — general usage.
• examples/safety-gate.ts — agent safety-gate starting point.

Suggested next test

For 20–100 real examples, run both this INT8 model and the FP32 reference. Compare the chosen answer, probabilities, and whether the safety gate allows/reviews/blocks. If important decisions differ, keep FP32 or revise the policy and threshold.

Original checkpoint license: Apache-2.0. This INT8 conversion is derived from the original checkpoint.