multivector-ModernBERT-base-msmarco-kd-old

ColBERT ModernBERT-base distilled from BGE on MS MARCO

This is a Multi-Vector Encoder model finetuned from answerdotai/ModernBERT-base on the ms-marco-en-bge dataset using the sentence-transformers library. It maps sentences & paragraphs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: answerdotai/ModernBERT-base <!-- at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8 -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: MaxSim
• Supported Modality: Text
• Training Dataset:
  • ms-marco-en-bge
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 32, 'document_length': 180, 'do_query_expansion': True, 'attend_to_expansion_tokens': False, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': False})
  (2): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~']})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-ModernBERT-base-msmarco-kd")
# Run inference: each input becomes a list of per-token vectors (variable length).
sentences = [
    'The weather is lovely today.',
    "It's so sunny outside!",
    'He drove to the stadium.',
]
query_embeddings = model.encode_query(sentences)
document_embeddings = model.encode_document(sentences)
print(query_embeddings[0].shape)
# (num_query_tokens, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[31.8979, 31.0701, 30.5561],
#         [31.2099, 31.9224, 30.6434],
#         [30.6855, 30.6123, 31.8958]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Multi Vector Information Retrieval

• Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with <code>MultiVectorInformationRetrievalEvaluator</code>

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with <code>MultiVectorNanoBEIREvaluator</code> with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nq",
          "fiqa2018"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with <code>MultiVectorNanoBEIREvaluator</code> with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

ms-marco-en-bge

• Dataset: ms-marco-en-bge at ad24729
• Size: 20,000 training samples
• Columns: <code>query</code>, <code>documents</code>, and <code>scores</code>
• Approximate statistics based on the first 100 samples:

  	query	documents	scores
  type	string	list	list
  modality	text
  details	<ul><li>min: 4 tokens</li><li>mean: 9.92 tokens</li><li>max: 21 tokens</li></ul>	<ul><li>size: 32 elements</li></ul>	<ul><li>size: 32 elements</li></ul>

• Samples:

  query	documents	scores
  <code>define extreme</code>	<code>['extremist. 1 AN EXTREMIST IS SOMEONE WHO SUPPORTS AN IDEA, CAUSE, OR SET OF VALUES SO ADAMANTLY AND WITHOUT COMPROMISE THAT SAID PERSON WILL USE THEIR IDEAS TO JUSTIFY ANYTHING THEY DO.', "at the extreme end meaning, at the extreme end definition	English Cobuild dictionary. extreme. 1 adj Extreme means very great in degree or intensity. The girls were afraid of snakes and picked their way along with extreme caution., ...people living in extreme poverty., ...the author's extreme reluctance to generalise.", 'extremity (plural extremities) 1 The most extreme or furthest point of something. 2 An extreme measure. 3 A hand or foot. A limb (major appendage of human or animal such as a leg an arm or a wing)', ': extreme in a way that is not normal or that shows an illness or mental problem. medical: relating to or caused by disease.: of or relating to the study of diseases: relating to pathology. extreme in a way that is not normal or that shows an illness or mental problem. medical: relating to or caused by disease.', 'Definition of extreme. 1a : existing in a very high degree extreme povertyb : going to great or exaggerated lengths : radical went on an extreme dietc : exceeding the ordinary, usual, or expected extreme weather conditions. 2 archaic : last.', ...]</code>
  <code>what does chattel mean on credit history</code>	<code>["Duhaime's Law Dictionary. Chattel Mortgage Definition: Related Terms: Chattel, Mortgage. When a lien is given on goods, chattels, moveable or personal property (other than real property in which case it is referred to as just a mortgage), in writing, to guarantee the payment of a debt or the execution of some action.", 'From Wikipedia, the free encyclopedia. Chattel mortgage, sometimes abbreviated CM, is the legal term for a type of loan contract used in some states with legal systems derived from English law. Under a typical chattel mortgage, the purchaser borrows funds for the purchase of movable personal property (the chattel) from the lender. The lender then secures the loan with a mortgage over the chattel.', 'Chattel Mortgages In Detail. A Chattel Mortgage uses your vehicle or some other (non-real estate) property as the security on the loan meaning you can access a low interest rate.ncidentally, these loans can be used for other purposes such as business equipment. If you have a preference for Chattel Mortgage, ask the team at 360 Finance. Term / Length of the loan â\x80\x93 the life of the loan or the time you have to pay it off.', 'A chattel mortgage is a mortgage that provides for a security interest in assets other than real estate to secure the loan. In the event of a default in payments, the lender has a lien in the assets used as collateral for the loan. In most states, a security agreement has replaced the use of chattel mortgages. chattel mortgage is a mortgage that provides for a security interest in assets other than real estate to secure the loan. In the event of a default in payments, the lender has a lien in the assets used as collateral for the loan. In most states, a security agreement has replaced the use of chattel mortgages.', 'A Chattel Mortgage is a type of loan contract that allows the buyer to take ownership of a vehicle at the time of purchase. The lender provides the buyer with the total loan amount to cover the price of the vehicle (chattel) so that it can be bought outright.', ...]</code>	<code>[0.7124203443527222, 0.7379189729690552, 0.5786551237106323, 0.6142299175262451, 0.6755089163780212, ...]</code>
  <code>what was the great leap forward brainly</code>	<code>['It was a clever scheme that was hatched soon after the 1949 revolution. The first phase was to send spies to the west during the great leap forward in the 1950s to plant falsified basic science into our western understanding of physics.', 'Great Leap Forward Devolution Into the Great Famine . Yang Jisheng, the author of Tombstone , wrote in the New York Times, â\x80\x9cThe Great Leap Forward that Mao began in 1958 set ambitious goals without the means to meet them. A vicious cycle ensued; exaggerated production reports from below emboldened the higher-ups to set even loftier targets.', 'In 1958 Mao introduced a second five year plan which became known as the â\x80\x98Great Leap Forwardâ\x80\x99 (GLF). He believed it was possible for China to overtake Britain as a leading industrial power within seven years and the USA soon after.n 1958 Mao introduced a second five year plan which became known as the â\x80\x98Great Leap Forwardâ\x80\x99 (GLF). He believed it was possible for China to overtake Britain as a leading industrial power within seven years and the USA soon after.', 'The Great Leap Forward approach was epitomized by the development of small backyard steel furnaces in every village and urban neighbourhood, which were intended to accelerate the industrialization process.', 'The Great Leap Forward was begun in 1957 by Chairman Mao Zedong to bring the nation quickly into the forefront of economic development. Mao wanted China to become a leading industrial power, and to accomplish his goals he and his colleagues pushed for the construction of steel plants across the country.', ...]</code>	<code>[0.6462352871894836, 0.7880821228027344, 0.791019856929779, 0.7709633111953735, 0.8284491300582886, ...]</code>

• Loss: <code>MultiVectorDistillKLDivLoss</code> with these parameters:

  {
      "score_metric": "colbert_kd_scores",
      "normalize_scores": true,
      "temperature": 1.0,
      "size_average": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 4
• num_train_epochs: 1
• learning_rate: 3e-05
• warmup_steps: 0.05
• bf16: True
• per_device_eval_batch_size: 4
• load_best_model_at_end: True

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 4
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 3e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 4
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary>

• The bold row denotes the saved checkpoint.

</details>

Training Time

• Training: 1.2 hours
• Evaluation: 27.5 minutes
• Total: 1.7 hours

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.6.0.dev0
• Transformers: 5.8.0.dev0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0.dev0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Additional Resources

• Sentence Transformers Documentation: the full documentation site, including training, evaluation, and pre-trained model catalogs.
• PyLate: the upstream library whose features were absorbed into Sentence Transformers for multi-vector / late-interaction models.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultiVectorDistillKLDivLoss

@inproceedings{santhanam-etal-2022-colbertv2,
    title = "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction",
    author = "Santhanam, Keshav and Khattab, Omar and Saad-Falcon, Jon and Potts, Christopher and Zaharia, Matei",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    year = "2022",
    publisher = "Association for Computational Linguistics",
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->