finance-minilm-l6-v2

finance-minilm-l6-v2

A vocabulary-expanded version of sentence-transformers/all-MiniLM-L6-v2 adapted to the finance domain via embedding surgery.

The base model's 30,522-token WordPiece vocabulary splits core finance terminology into subword fragments (e.g. securitization → sec ##uri ##ti ##zation). This model adds 15 finance terms as single tokens with their own embeddings, leaving everything else untouched.

Added tokens

securitization, collateralized, counterparty, underwriting, hedging, liquidity, volatility, amortization, refinancing, receivables, covenants, tranche, escrow, annuity, actuarial

Method

1. The 15 terms were selected by training a 10k WordPiece vocabulary on a finance corpus, diffing it against the base vocabulary, and hand-picking distinctly financial whole words from the 4,398 missing tokens.
2. tokenizer.add_tokens(...) expanded the vocabulary; model.resize_token_embeddings(...) grew the embedding matrix.
3. Pretrained embeddings are preserved exactly — all 30,522 original rows are bit-for-bit identical to the base model (verified against a pre-surgery snapshot).
4. Each new token's embedding is initialized as the mean of the embeddings of the subword pieces it previously split into, placing it in a meaningful region of the embedding space rather than at a random point.

No further fine-tuning has been applied; the new embeddings are initializations intended as a starting point for domain fine-tuning.

Usage

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("adammoood/finance-minilm-l6-v2")
model = AutoModel.from_pretrained("adammoood/finance-minilm-l6-v2")
model.eval()

sentences = ["The bank improved its liquidity through securitization of receivables."]
enc = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
with torch.no_grad():
    out = model(**enc)

# Mean pooling + normalization, as in the base model
mask = enc["attention_mask"].unsqueeze(-1).float()
embeddings = F.normalize((out.last_hidden_state * mask).sum(1) / mask.sum(1), dim=1)

tokenizer.tokenize("The tranche was collateralized to reduce counterparty risk.")
# ['the', 'tranche', 'was', 'collateralized', 'to', 'reduce', 'counterparty', 'risk', '.']