Onnx-Library

ONNX-Library

ONNX exports of my SpikeWhale-DNA language models (Byrne / Escarda), Byrne ASR and TTS, and the Byrne-VLM vision encoder. Exported from the original PyTorch weights with the legacy TorchScript exporter (opset 17). Parity-verified against the source model.

Each model is its own folder. The .onnx graphs are the network only - tokenizers, text frontends, and CTC/beam decoders stay in the source repos (linked below), same as at inference.

Contents

Language models - SpikeWhaleLM (13)

input_ids [B,T] (int64) → logits [B,T,16512] · dynamic batch & sequence · file model.onnx (~383 MB each)

Tokenizer (tokenizer.json + spike_tokenizer.py) is in each source repo. Verified 100 % argmax-token agreement with PyTorch (the deep custom ops add ~1e-1 fp noise to the wide logits - harmless; base variants are near-exact).

Speech recognition - Byrne-ASR-English/model.onnx (~50 MB)

mel [B,80,T] (float32) → logits [B,T',29] (CTC, dynamic frames). Mel frontend params: sample_rate 24000, n_fft 1024, hop 256, n_mels 80, log-mel. Vocab: <blank>, space, a-z, '. The lexicon / bigram / ARPA beam-search decode lives in the source repo (Byrne-ASR-English). Parity 7e-6.

Vision encoder - Byrne-VLM-131M/vision.onnx (~167 MB)

image [B,3,448,448] (float32, [-1,1]) → pooled [B,512] + tokens [B,784,512]. ViT-style, patch 16, native 448×448 (28×28 patch grid), 2D axial RoPE. Source: Byrne-VLM-131M. Parity 1e-6. (The multimodal LM half is not included here.)

Text-to-speech - Byrne-Speech/ (2-stage, ~49 MB)

1. acoustic.onnx - FastSpeech2: ids [B,Tp] (int64) + plen [B] → mel [B,80,Tm] (variable length; length regulator generalizes across text lengths).
2. vocoder.onnx - HiFi-GAN: mel [B,80,Tm] → wav [B,1,Tm*256] (24 kHz, hop 256). weight_norm folded.

Char text frontend (text_to_char_sequence) is in the source repo (Byrne-Speech). Chain: text → acoustic.onnx → mel → vocoder.onnx → wav. Parity 7e-7.

Tools (4)

Vision-language generation (full pipeline)

Byrne-VLM-131M (captioning) and Byrne-Docling-131M (document -> DocTags) are generative - each ships THREE files for real image->text generation (the single vision.onnx is the encoder only):

• vision_connector.onnx : image[1,3,448,448] -> image_embeds[1,784,640] (vision encoder + projector)
• lm_decode.onnx : inputs_embeds[1,T,640] -> logits[1,T,V] (LoRA-applied LM)
• embed_tokens.npy : [V,640] token-embedding table (for generated text tokens)

Generate: encode the image once, feed image_embeds as the prefix, then autoregressively append embed_tokens[next_token] and re-run lm_decode:

import numpy as np, onnxruntime as ort
va=ort.InferenceSession("Byrne-VLM-131M/vision_connector.onnx")
lm=ort.InferenceSession("Byrne-VLM-131M/lm_decode.onnx")
emb=np.load("Byrne-VLM-131M/embed_tokens.npy")
ie=va.run(["image_embeds"],{"image":img})[0]          # img: [1,3,448,448] float32 in [-1,1]
out=[]
for _ in range(48):
    x = ie if not out else np.concatenate([ie, emb[out][None]], 1)
    nxt = int(lm.run(["logits"],{"inputs_embeds":x})[0][0,-1].argmax())
    if nxt==EOS: break
    out.append(nxt)                                    # decode with the model's tokenizer

Verified to produce PyTorch-identical greedy output (Byrne-VLM: "A group of people standing on the ground."). Byrne-VLM decodes with the shared LM tokenizer; Byrne-Docling with its source tokenizer_doctags.json.

Usage (ONNX Runtime)

Language model (needs the tokenizer from the source repo):

import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("Byrne-86M/model.onnx", providers=["CPUExecutionProvider"])
input_ids = np.array([[1, 23, 45, 6]], dtype=np.int64)          # from the SpikeWhale tokenizer
logits = sess.run(["logits"], {"input_ids": input_ids})[0]      # [1, T, 16512]
next_id = logits[0, -1].argmax()                                # greedy next token

TTS (chain the two graphs):

import onnxruntime as ort, numpy as np
ac = ort.InferenceSession("Byrne-Speech/acoustic.onnx", providers=["CPUExecutionProvider"])
vo = ort.InferenceSession("Byrne-Speech/vocoder.onnx",  providers=["CPUExecutionProvider"])
ids  = np.array([[...]], dtype=np.int64)                         # text_to_char_sequence(text)
plen = np.array([ids.shape[1]], dtype=np.int64)
mel = ac.run(["mel"], {"ids": ids, "plen": plen})[0]            # [1,80,Tm]
wav = vo.run(["wav"], {"mel": mel})[0]                          # [1,1,Tm*256] @ 24 kHz

Vision encoder:

pooled, tokens = sess.run(["pooled","tokens"], {"images": img})  # img [B,3,448,448] float32 in [-1,1]

All graphs use dynamic batch (and dynamic sequence/frames where noted), so batching works out of the box.

Provenance

Exported 1:1 from the PyTorch checkpoints in the linked source repos (opset 17, dynamo=False), each verified against its original model. License: Apache-2.0.