specter2-proximity-onnx

SPECTER2 + Proximity Adapter (ONNX)

This is an ONNX export of allenai/specter2_base with the proximity adapter fused in. It produces 768-dimensional embeddings optimized for scientific document similarity and citation prediction.

The adapter weights are baked into the model graph — no separate adapter loading needed.

Usage

import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

# Download model and tokenizer
model_path = hf_hub_download("adamlabadorf/specter2-proximity-onnx", "specter2_proximity.onnx")
data_path = hf_hub_download("adamlabadorf/specter2-proximity-onnx", "specter2_proximity.onnx.data")
tokenizer_path = hf_hub_download("adamlabadorf/specter2-proximity-onnx", "tokenizer/tokenizer.json")

# Load
session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
tokenizer = Tokenizer.from_file(tokenizer_path)
tokenizer.enable_truncation(max_length=512)
tokenizer.enable_padding(pad_id=0, pad_token="[PAD]")

# Encode
texts = ["Effect of gut microbiota on neuroinflammation"]
encodings = tokenizer.encode_batch(texts)
input_ids = np.array([e.ids for e in encodings], dtype=np.int64)
attention_mask = np.array([e.attention_mask for e in encodings], dtype=np.int64)

embeddings = session.run(None, {"input_ids": input_ids, "attention_mask": attention_mask})[0]
# Normalize
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)

Model Details

• Base model: allenai/specter2_base (SciBERT)
• Adapter: Proximity adapter from allenai/specter2 — 12 bottleneck layers (768→48→768 with ReLU)
• Embedding dimension: 768
• Max sequence length: 512 tokens (SciBERT wordpiece tokenizer)
• ONNX opset: 17
• Export validation: Max absolute difference vs PyTorch < 4e-6

Files

• specter2_proximity.onnx — ONNX graph structure
• specter2_proximity.onnx.data — Model weights (external data)
• tokenizer/ — SciBERT tokenizer files

Why ONNX?

This export allows running SPECTER2 embeddings without PyTorch or transformers as runtime dependencies — only onnxruntime and tokenizers are needed. Useful for lightweight deployments and applications that already use ONNX Runtime.

Citation

If you use this model, please cite the original SPECTER2 paper:

@inproceedings{singh2023scirepeval,
  title={SciRepEval: A Multi-Format Benchmark for Scientific Document Representations},
  author={Singh, Amanpreet and D'Arcy, Mike and Cohan, Arman and Downey, Doug and Feldman, Sergey},
  booktitle={EMNLP},
  year={2023}
}

License

Apache 2.0, consistent with the original allenai/specter2 model.