fin-sparse-encoder-doc-v1-onnx

fin-sparse-encoder-doc-v1-onnx

ONNX + INT8 quantized version of oneryalcin/fin-sparse-encoder-doc-v1 for CPU-efficient document encoding.

This is the document encoder path only — it produces sparse SPLADE vectors for indexing financial documents (SEC filings, earnings call transcripts). Query encoding uses a separate IDF lookup table (sub-ms, no neural model needed).

Model Variants

Performance

Domain Evaluation (Financial Documents)

The parent model (fin-sparse-encoder-doc-v1) was evaluated on 2,028 held-out financial test examples:

Inference Latency (seq_len=512, 1 thread)

Benchmarked on Apple M-series CPU. Server CPUs with AVX512-VNNI will see larger INT8 speedups (~2-3x).

Usage

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

# Load
tokenizer = AutoTokenizer.from_pretrained("oneryalcin/fin-sparse-encoder-doc-v1-onnx")
sess = ort.InferenceSession("model_quantized.onnx", providers=["CPUExecutionProvider"])

# Encode a document
text = "Revenue increased 12% year over year to $4.2 billion in Q4 2023."
inputs = tokenizer(text, return_tensors="np", padding="max_length", max_length=512, truncation=True)
logits = sess.run(None, {"input_ids": inputs["input_ids"], "attention_mask": inputs["attention_mask"]})[0]

# SpladePooling: log1p_relu activation (matches OpenSearch v3 models)
masked = logits * inputs["attention_mask"][..., None]
pooled = masked.max(axis=1)
sparse_vector = np.log1p(np.log1p(np.maximum(pooled, 0.0)))  # [1, 30522]

# Convert to token->weight dict (for inverted index)
nonzero = np.nonzero(sparse_vector[0])[0]
token_weights = {tokenizer.decode([tid]): float(sparse_vector[0, tid]) for tid in nonzero}
print(f"Active dimensions: {len(token_weights)}")
print(f"Top tokens: {sorted(token_weights.items(), key=lambda x: -x[1])[:10]}")

Architecture

Input text
  → Tokenizer (max_length=512)
  → ONNX model (MLM logits) [batch, seq, 30522]
  → SpladePooling: log(1 + log(1 + ReLU(max_over_seq(logits * mask))))
  → Sparse vector [batch, 30522]

Base model: opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte (Alibaba-NLP/new-impl architecture).

Fine-tuned on financial-filings-sparse-retrieval-training (18K examples, 2 epochs).

Export Details

• Exported via torch.onnx.export (legacy tracer, opset 17)
• INT8: dynamic quantization via onnxruntime.quantization.quantize_dynamic (per-channel, QInt8)
• Numerical verification: FP32 ONNX max diff vs PyTorch = 0.000057