ovos-m2v-intents-en

OVOS intent classifier, English

Static-embedding intent classifiers for OpenVoiceOS, trained on English only. Each maps an utterance to one of 217 intent labels of the form <skill_id>:<intent_name>, the names the OVOS m2v pipeline registers at runtime.

The repository holds one model for each size of the potion English base. The root holds the most accurate one, on potion-base-32M. Inference is a vector lookup, an average and one linear layer, so every size runs on a CPU.

Which model

For comparison, the multilingual model ovos-m2v-intents-multilingual scores 0.8400 on all 1,506 English gold rows at 532 MB.

The int8 folder is the best choice when size matters: it loses little against the root model at a quarter of its size, and beats potion-base-8M at about the same size. The root model has an ONNX copy in onnx/, and each smaller base one in <folder>/onnx/.

Use

The OVOS m2v pipeline loads the multilingual model by default. To use this one for English, set models (or model for a device that only speaks English), and subfolder for any folder other than the root:

{"intents": {"ovos-m2v-pipeline": {
  "models": {"en": "OpenVoiceOS/ovos-m2v-intents-en"},
  "subfolder": "int8"}}}

subfolder applies to every model the pipeline loads, so set it only when all of them have that folder.

In Python, download one folder and load it:

from huggingface_hub import snapshot_download
from model2vec.inference import StaticModelPipeline

path = snapshot_download("OpenVoiceOS/ovos-m2v-intents-en", allow_patterns=["int8/*"])
model = StaticModelPipeline.from_pretrained(f"{path}/int8")
model.predict(["set a timer for ten minutes"])

For the root model, StaticModelPipeline.from_pretrained("OpenVoiceOS/ovos-m2v-intents-en").

ONNX

onnx/model.onnx runs without model2vec. It takes input_ids and attention_mask (int64, batch by tokens) and returns probabilities (float32, batch by 217), in the order of head_config.classes in onnx/config.json. Tokenize with onnx/tokenizer.json without special tokens, drop the unknown token, and keep at most 512 tokens:

import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("onnx/tokenizer.json")
classes = json.load(open("onnx/config.json"))["head_config"]["classes"]
unk = tok.token_to_id("[UNK]")
ids = [i for i in tok.encode("set a timer for ten minutes", add_special_tokens=False).ids if i != unk][:512]
sess = ort.InferenceSession("onnx/model.onnx")
probs = sess.run(None, {"input_ids": np.array([ids]), "attention_mask": np.ones((1, len(ids)), np.int64)})[0]
print(classes[probs[0].argmax()])

On 1,513 sentences each ONNX copy gives the same top label as its model2vec model.

The previous model

The previous English model, on potion-base-8M with 209 labels in the older label names, is revision f67cb4bcc411e10658a3cf00db8fdaf203791341. Set "revision" to that value to load it. On the 637 English gold rows whose label both models know (83 labels), the root model here scores 0.8823 and the previous model 0.8226.

Training

Training used 8,020 templates, 265,979 rows; 825 templates, 25,578 rows, were held out. The held-out accuracy is 0.9985 for potion-base-32M, 0.9925 for 8M, 0.9905 for 4M and 0.9895 for 2M.

Results

The gold sentences are the skills' own golden utterances, not generated from the templates the models train on. 1,506 are English; 6 have a label with no English training row and are not scored, which leaves 1,500. 1,028 of them are flagged needs_manual and are scored like the others.

The settings (row cap, minimum, epochs, base) were chosen on one half of the English gold rows, so the gold figures above are slightly optimistic.

Limits

• The gold sentences are model-written. Every gold sentence was written by a language model. needs_manual marks rows meant for a human check, and whether that check happened is not recorded, so the gold figures measure agreement with model-written sentences, not with recorded users.
• Classifier mode only. These models are measured as classifiers. Prototype mode in the OVOS pipeline was not measured with them.

License

Apache-2.0. The base models are MIT.