bert-cord-27m-mlm-onnx

BERT-Cord 27M — ONNX MLM baseline

Package version: 0.1.2-hf-onnx · Future HF repo: sikkha/bert-cord-27m-mlm-onnx

Purpose

A portable, framework-neutral ONNX Runtime inference artifact for the custom BertForMaskedLM encoder from the bert_cord research project. It performs masked-token prediction only.

Honest scope. This is a synthetic MLM development checkpoint. It is not yet a "mini-amygdala" coordinator. It performs no coordination, routing, memory activation, task lifecycle control, consciousness, or general language understanding, and is not production ready. No natural-language tokenizer is bundled — inputs are integer token ids.

Development stage

Milestone 0.7 (ONNX export). Derived, inference-only artifact; the PyTorch checkpoint in the source repository remains the training source of truth. Not a Transformers AutoModel — do not assume transformers compatibility.

Architecture summary

Custom from-scratch BERT encoder (not Hugging Face BertModel): learned token/position/ token-type embeddings, 8 post-LN transformer layers, 6 attention heads, hidden size 384, intermediate size 1536, tied input/output embeddings, GELU FFN.

• Exact parameter count: 27,010,304
• Task: masked-language modeling (MLM)
• ONNX opset: 18
• Precision: FP32

Input / output contract

batch and sequence are dynamic axes.

External data requirement

The weights are stored externally: you must download both onnx/model.onnx (graph) and onnx/model.onnx.data (weights, ~102.5 MB) and keep them side by side. Total artifact ≈ 103.1 MB. Loading model.onnx without model.onnx.data will fail.

Installation

pip install -r requirements.txt   # numpy>=1.24, onnxruntime>=1.17

Standalone inference example

python inference.py

Or in code:

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("onnx/model.onnx", providers=["CPUExecutionProvider"])
seq = [1, 50, 67, 7, 50, 67, 3, 50, 67, 7, 2]           # ids; 3 == [MASK]
ii = np.array([seq], dtype=np.int64)
am = np.ones_like(ii); tt = np.zeros_like(ii)
logits = sess.run(["logits"], {"input_ids": ii, "attention_mask": am,
                                "token_type_ids": tt})[0]
print(logits.shape)                                       # (1, seq, 32000)

Parity results (this package)

Packaged ONNX vs the source PyTorch model, identical deterministic inputs, FP32 CPU:

• max absolute logit difference: 6.20e-06 (tolerances rtol=1e-3, atol=2e-3)
• mean absolute difference: 8.25e-07 – 8.52e-07
• masked-position top-5 agreement: 1.00
• NaN/Inf: False
• dynamic axes: True (batch [1, 2, 3] × seq [64, 128])
• ONNX structural check: PASS

Validated vs unvalidated runtimes

• Validated: CPUExecutionProvider, FP32.
• Unvalidated: CUDAExecutionProvider (onnxruntime-gpu), Apple CoreML, and FP16 / BF16. No claim is made about them.

Intended uses

Portable CPU inference / benchmarking of a small custom encoder graph; a reference artifact for the bert_cord research line; ONNX Runtime integration experiments.

Out-of-scope uses

Any language-understanding, chat, generation, coordination/routing, safety-critical, or production use. This model has no such capability.

Limitations

Synthetic MLM baseline trained on a learnable copy-motif corpus (not natural language). Outputs are only meaningful as an MLM-graph sanity signal. No tokenizer, no semantics.

Training-data disclosure

Trained on synthetic token sequences (a tiled copy-motif corpus) for pipeline validation — no natural-language corpus, no private data.

Tokenizer disclosure

No tokenizer is bundled. Inputs are raw integer token ids. Reserved ids: PAD=0, CLS=1, SEP=2, MASK=3, UNK=4; "real" tokens start at id 5.

Reproducibility & provenance

Model provenance (the commit the ONNX model was exported from) is tracked separately from packaging provenance (the commit of the tooling that assembled this package):

• Source repository: https://github.com/sikkha/bert-cord
• Model source commit: 0e17db558ebcce29f40b49d546af8b2704640230
• Model source tag: v0.1.1-onnx
• Packaging source commit: c3b9d28fa9f7333c2abaf02a451249c8f5aa03d8
• Packaging source tag: v0.1.2-hf-package
• Exported with PyTorch's ONNX exporter (opset 18); parity re-measured at package-build time.

Package file layout

bert-cord-27m-mlm-onnx/
├── README.md          (this model card)
├── LICENSE            (Apache-2.0, copied from source)
├── config.json        (architecture + provenance)
├── evaluation.json    (measured parity + checksums)
├── requirements.txt   (numpy, onnxruntime)
├── inference.py       (standalone ONNX Runtime example)
├── MANIFEST.json      (files + sizes + SHA-256)
└── onnx/
    ├── model.onnx       (graph; references model.onnx.data)
    └── model.onnx.data  (external FP32 weights — download together)

Citation

@software{bert_cord_27m_mlm_onnx,
  title  = {BERT-Cord 27M — ONNX MLM baseline},
  author = {Kan (sikkha)},
  year   = {2026},
  url    = {https://github.com/sikkha/bert-cord},
  note   = {Synthetic MLM development baseline; model commit 0e17db558ebcce29f40b49d546af8b2704640230}
}

License

Apache-2.0 (see LICENSE).