MarketBERT

Market2Vec

Trademark-Based Product Timeline Embeddings (Forecasting MLM)

This repo builds and trains Market2Vec from trademark data using a sequence-of-products view:

• Firms (owners) are treated as entities with a timeline of products
• Each product is a sequential event
• Each product has an item set (goods/services descriptors) treated as a basket (set, not order)

---

Two training versions (two objectives)

We support two versions of the MLM objective:

Version A — Forecasting (last-event prediction)

Purpose: learn to forecast the last product’s items from the firm’s earlier product history.

• We identify the last product event in the sequence: the segment between the last [APP] and the next [APP_END] (or [SEP])
• We force-mask ITEM_* tokens inside that last event (mask probability = 1.0)
• (Optional) random masking elsewhere can be turned off for “clean forecasting” evaluation

This version is best when your downstream use-case is “given past trademark products, predict items in the most recent/next product”.

Version B — Random MLM over the full product sequence

Purpose: learn general co-occurrence/semantic structure of items in firm timelines (classic MLM).

• We mask tokens randomly across the whole sequence with probability p (e.g., 15%)
• This includes items across all product events, not only the last one
• This version behaves like standard BERT MLM, but applied to your product timeline format

This version is best when you want broad embeddings capturing item relationships and temporal context without specifically focusing on forecasting the last event.

---

What the forecasting masking means (in practice)

A packed firm sequence looks like:

[CLS] DATE_YYYY_MM [APP] NICE_* ... ITEM_* ... [APP_END] DATE_YYYY_MM [APP] NICE_* ... ITEM_* ... [APP_END] ... DATE_YYYY_MM [APP] NICE_* ... ITEM_* ... [APP_END] <-- last event [SEP]

• Version A: masks ITEM_* in the last [APP]..[APP_END] segment (forecasting target)
• Version B: masks tokens randomly across the entire sequence (classic MLM)

---

Metrics

Validation reports:

• AccAll: accuracy over all masked tokens
• Item@K: Top-K accuracy restricted to masked positions where the true label is an ITEM_* token

For forecasting, Item@K is the main metric because it directly measures how well the model predicts items in the last product basket.

---

Results — MarketBERT (pretrained Market2Vec checkpoint)

Training Summary

• Model: A4_full_fixed_alpha_optionA_h512_h32
• Best validation loss: 3.6433

Validation (HARD)

• Acc@1: 0.5996
• Acc@5: 0.6651
• Acc@10: 0.6944

---

Usage (Hugging Face)

from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("HamidBekam/MarketBERT")
model = AutoModel.from_pretrained("HamidBekam/MarketBERT")