laya-onnx

Laya — ONNX (fp16)

Laya exported to ONNX, so it runs under ONNX Runtime — on a CPU, a GPU, or in a browser — with no PyTorch installed. This is the default build.

Laya is not a generative model. Give it a state (text, email, ticket, JSON) and typed questions, and it returns one calibrated distribution per question in a single forward pass. Nothing is sampled, so the answer is always one of the options you supplied.

pip install onnxruntime huggingface_hub tokenizers numpy
hf download inferenceprince/laya-onnx --local-dir laya-onnx

The weights are the same size as the base checkpoint's model.safetensors (842.6 MB) because they are the same fp16 tensors. The 3.3 MB graph file is the only addition — safetensors carries no architecture, so the structure lives in the authors' Python; ONNX has to serialise it. That is what buys you a runtime with no framework dependency.

How to use it

The graph takes five inputs and returns logits and act_logits. Everything else — the prompt format, the tokeniser, the temperature — is in the two files beside it. This is the whole thing:

import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

MODEL_DIR = "laya-onnx"

config = json.load(open(MODEL_DIR + "/rl_agent_config.json"))
tokenizer = Tokenizer.from_file(MODEL_DIR + "/tokenizer/tokenizer.json")
session = ort.InferenceSession(MODEL_DIR + "/model.onnx", providers=["CPUExecutionProvider"])

CLS = tokenizer.token_to_id("[CLS]")
SEP = tokenizer.token_to_id("[SEP]")
MASK = tokenizer.token_to_id("[MASK]")


def tokenize(text):
    """Text to a list of token ids."""
    return tokenizer.encode(text, add_special_tokens=False).ids


def temperature_for(question_type, option_count):
    """The fitted temperature for this kind of question."""
    if option_count <= 2:
        size = "2"
    elif option_count <= 5:
        size = "3-5"
    elif option_count <= 10:
        size = "6-10"
    else:
        size = "11+"

    key = question_type + ":" + size
    if key in config["temperature_by_options"]:
        return config["temperature_by_options"][key]
    return config["temperature"][0]


# The question we want answered.
state = "I was billed twice. Please refund the duplicate."
question = "Which team should handle this?"
options = {
    "billing": "invoices, payments, refunds",
    "technical": "bugs, outages",
    "sales": "pricing",
}

# Build the prompt. The model reads:
#   [CLS] choice question: <question> [SEP]
#   [MASK] billing: ...  [MASK] technical: ...  [MASK] sales: ... [SEP]
#   <the text to analyse> [SEP]
# Every option gets its own [MASK] token, and the model scores that position.
token_ids = [CLS] + tokenize("choice question: " + question) + [SEP]

option_positions = []          # where each option's [MASK] sits in token_ids
for label, description in options.items():
    option_positions.append(len(token_ids))
    token_ids.append(MASK)
    token_ids += tokenize(" " + label + ": " + description)[:48]

token_ids.append(SEP)

# Whatever room is left in the 512-token window goes to the text being analysed.
room = config["max_len"] - len(token_ids) - 1
token_ids += tokenize(state)[:room]
token_ids.append(SEP)

# Run the model. It returns one raw score per option.
option_count = len(option_positions)
outputs = session.run(None, {
    "input_ids":      np.array([token_ids], dtype=np.int64),
    "attention_mask": np.ones((1, len(token_ids)), dtype=np.int64),
    "marker_pos":     np.array([option_positions], dtype=np.int64),
    "marker_mask":    np.ones((1, option_count), dtype=bool),
    "qtype":          np.array([0], dtype=np.int64),   # 0=choice, 1=score, 2=noul
})
logits = outputs[0]

# Turn scores into probabilities. The graph returns raw scores; dividing by the
# fitted temperature is what makes them meaningful, so this step is not optional.
temperature = temperature_for("choice", option_count)
scores = logits[0, :option_count] / temperature

scores = scores - scores.max()          # keep the numbers small before exponentiating
probabilities = np.exp(scores)
probabilities = probabilities / probabilities.sum()

# Print the winner and the full distribution.
labels = list(options.keys())
print("answer:", labels[int(probabilities.argmax())])
for label, probability in zip(labels, probabilities):
    print("  %-12s %.4f" % (label, probability))

Real output from running the above against this build:

answer: billing
  billing      0.9696
  technical    0.0152
  sales        0.0152

Notes that matter:

• Options must fit head_max_len (192 tokens). The instructions share that budget with the options, and options are individually capped at 48 tokens. Long labels get silently truncated — keep choice under ~20 options.
• marker_pos must be an int64 array of length ≥ the number of options, and marker_mask must be True for the real ones. Pad both if you batch.
• Divide the logits by the temperature before softmax. The graph returns raw logits; the fitted values in rl_agent_config.json are what make the probabilities meaningful.
• Batching: rows are independent, so several questions can share one forward pass. Each row needs its own marker_pos, marker_mask and qtype.

Startup

Measured on an Intel i5-14400F (6 performance + 4 efficiency cores), CPU only, from a cold process:

Startup is the largest difference between the two, and it is stable across runs. Inference latency is broadly comparable — in the quietest measurements roughly 150 ms for one short question against 200 ms for PyTorch — but this machine is shared and load swung identical work by more than 10x between runs, so treat latency as indicative rather than as a specification.

Accuracy

116 synthetic items across 15 categories. The labels were generated by a language model, not written by human annotators. So these figures measure agreement with a model-authored rubric rather than independent accuracy, and any shared bias between the labels and the model under test would flatter the result. Treat them as a smoke test that a build behaves like the reference, and validate on your own data before relying on the numbers.

Fidelity: the same answer as the upstream PyTorch model on all 116 items. Worst probability shift 1.8e-03. Tokenisation and prompt construction are byte-identical to the reference.

Where it works: classification with genuinely distinct categories — routing, prompt-injection detection, refund/churn/phishing detection, agent-trace failure labelling. 87–100% on those.

Where it does not: fine procedural distinctions (invoice hold vs request_info vs reject: 25%) and abstract severity ladders (16.7%). Ordinal score is coarse — never wrong by more than one level, but only half the time exactly right. choice confidence reads far lower than its real accuracy, so recalibrate on your own data before automating on it.

Limits

• English only. Collapses on non-Latin scripts while staying confident, so confidence gating will not catch it. Use laya-multilingual for anything else.
• Not a zero-shot decision engine. The authors state the base checkpoints are near chance on their own typed-decisions benchmark; accuracy comes from fine-tuning on your data.
• Keep choice under ~20 options — options share a fixed 192-token prompt budget.
• act_probability carries no signal — it saturates near 1.0 on every input tested.

Attribution

Model, training and weights by Nandakishor M and Convai Innovations, Apache-2.0. This is an independent ONNX conversion, not an official Convai Innovations release.