reranker-msmarco-v1.1-ettin-encoder-17m-bce

CrossEncoder based on jhu-clsp/ettin-encoder-17m

This is a Cross Encoder model finetuned from jhu-clsp/ettin-encoder-17m on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: jhu-clsp/ettin-encoder-17m <!-- at revision 987607455c61e7a5bbc85f7758e0512ea6d0ae4c -->
• Maximum Sequence Length: 7999 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

<!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("bansalaman18/reranker-msmarco-v1.1-ettin-encoder-17m-bce")
# Get scores for pairs of texts
pairs = [
    ['how to put word count on word', 'To insert a word count into a Word 2013 document, place the cursor where you would like the word count to appear (say in the Header or Footer) and then: 1  click the Insert tab. 2  click the Quick Parts icon (towards the right hand end of the toolbar). 3  on the drop down that appears, select Field...'],
    ['what is the difference between discipleship and evangelism', 'Discipleship, on the other hand, meant helping someone who was already a believer walk out the life of faith. The word “discipleship” brought to my mind a small group Bible study, a conversation across the table with another woman, or an accountability group. And I knew which one I preferred. As a result, the discipleship I offered others contained a lot of good information but lacked the transforming power that can only come from the gospel. (I was also, simply, a coward.). I am beginning to see that evangelism and discipleship are not all that different.'],
    ['what metal is a trophy made from', 'The trophy stands 36.5 centimetres (14.4 inches) tall and is made  of 5 kg (11 lb) of 18 carat (75%) gold with a base (13 centimetres  [5.1 inches] in diameter) containing t … wo layers of malachite. Making the world better, one answer at a time. Trophies can be made out of anything you want. however, aluminum is a very reliable and trustworthy metal and it.......... oh crap.......... i have to do a poo...'],
    ['how do you define what a cult is?', 'The term cult has been misused. The word cult comes from the French cult which is from the Latin word cultus (care/adoration) and Latin Colere (to cultivate.) So, we can plant seeds of good or bad. You can have political cults such as sit ins during the Vietnam War. A good cult could be a religious one, yet some Christians will consider Jehovah Witness a cult and have labeled them as preying on the weak. When someone labels such a thing it is usually because of the lack of understanding. Good cults are usually a small group of people that can have a cult in most anything.'],
    ['where is silchar', 'Silchar (/ˈsɪlˌʧə/ or /ˈʃɪlˌʧə/) (Bengali: শিলচর Shilchor) shilchôr is the headquarters Of cachar district in the state Of assam In. India it is 343 (kilometres 213) mi south east Of. Guwahati it is the-second largest city of the state in terms of population and municipal. area 1 The Bhubaneshwar temple is about 50 km from Silchar and is on the top the Bhuvan hill. 2  This is a place of pilgrimage and during the festival of Shivaratri, thousand of Shivayats march towards the hilltop to worship Lord Shiva.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'how to put word count on word',
    [
        'To insert a word count into a Word 2013 document, place the cursor where you would like the word count to appear (say in the Header or Footer) and then: 1  click the Insert tab. 2  click the Quick Parts icon (towards the right hand end of the toolbar). 3  on the drop down that appears, select Field...',
        'Discipleship, on the other hand, meant helping someone who was already a believer walk out the life of faith. The word “discipleship” brought to my mind a small group Bible study, a conversation across the table with another woman, or an accountability group. And I knew which one I preferred. As a result, the discipleship I offered others contained a lot of good information but lacked the transforming power that can only come from the gospel. (I was also, simply, a coward.). I am beginning to see that evangelism and discipleship are not all that different.',
        'The trophy stands 36.5 centimetres (14.4 inches) tall and is made  of 5 kg (11 lb) of 18 carat (75%) gold with a base (13 centimetres  [5.1 inches] in diameter) containing t … wo layers of malachite. Making the world better, one answer at a time. Trophies can be made out of anything you want. however, aluminum is a very reliable and trustworthy metal and it.......... oh crap.......... i have to do a poo...',
        'The term cult has been misused. The word cult comes from the French cult which is from the Latin word cultus (care/adoration) and Latin Colere (to cultivate.) So, we can plant seeds of good or bad. You can have political cults such as sit ins during the Vietnam War. A good cult could be a religious one, yet some Christians will consider Jehovah Witness a cult and have labeled them as preying on the weak. When someone labels such a thing it is usually because of the lack of understanding. Good cults are usually a small group of people that can have a cult in most anything.',
        'Silchar (/ˈsɪlˌʧə/ or /ˈʃɪlˌʧə/) (Bengali: শিলচর Shilchor) shilchôr is the headquarters Of cachar district in the state Of assam In. India it is 343 (kilometres 213) mi south east Of. Guwahati it is the-second largest city of the state in terms of population and municipal. area 1 The Bhubaneshwar temple is about 50 km from Silchar and is on the top the Bhuvan hill. 2  This is a place of pilgrimage and during the festival of Shivaratri, thousand of Shivayats march towards the hilltop to worship Lord Shiva.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with <code>CrossEncoderRerankingEvaluator</code> with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with <code>CrossEncoderNanoBEIREvaluator</code> with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 654,438 training samples
• Columns: <code>query</code>, <code>response</code>, and <code>label</code>
• Approximate statistics based on the first 1000 samples:

  	query	response	label
  type	string	string	float
  details	<ul><li>min: 11 characters</li><li>mean: 34.1 characters</li><li>max: 95 characters</li></ul>	<ul><li>min: 62 characters</li><li>mean: 417.26 characters</li><li>max: 939 characters</li></ul>	<ul><li>min: 0.0</li><li>mean: 0.13</li><li>max: 1.0</li></ul>

• Samples:

  query	response	label
  <code>what is a polyhedron definition</code>	<code>A polyhedron is said to be convex if its surface (comprising its faces, edges and vertices) does not intersect itself and the line segment joining any two points of the polyhedron is contained in the interior or surface. A polyhedron is a 3-dimensional example of the more general polytope in any number of dimensions. Polyhedra with congruent regular faces of six or more sides are all non-convex, because the vertex of three regular hexagons defines a plane. The total number of convex polyhedra with equal regular faces is thus ten, comprising the five Platonic solids and the five non-uniform deltahedra.</code>	<code>0.0</code>
  <code>what can you carry in hand luggage on easyjet</code>	<code>Each passenger who pays for a hold bag can take up to 20kg of luggage. This weight allowance applies to the passenger rather than to the bag so purchasing extra bags is possible but will not increase the weight allowance.</code>	<code>0.0</code>
  <code>what is dynamic segmentation in gis</code>	<code>The result of the dynamic segmentation process is a dynamic feature class known as a route event source. A route event source can serve as the data source of a feature layer in ArcMap. For the most part, a dynamic feature layer behaves like any other feature layer. Event locating errors. The dynamic segmentation process creates a shape for each row in the input route event table. In some cases, however, the shape of the event feature might be empty. This happens when there is a reason that the event can't be properly located.</code>	<code>0.0</code>

• Loss: <code>BinaryCrossEntropyLoss</code> with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: <code>query</code>, <code>response</code>, and <code>label</code>
• Approximate statistics based on the first 1000 samples:

  	query	response	label
  type	string	string	float
  details	<ul><li>min: 10 characters</li><li>mean: 33.73 characters</li><li>max: 117 characters</li></ul>	<ul><li>min: 57 characters</li><li>mean: 412.4 characters</li><li>max: 918 characters</li></ul>	<ul><li>min: 0.0</li><li>mean: 0.13</li><li>max: 1.0</li></ul>

• Samples:

  query	response	label
  <code>how to put word count on word</code>	<code>To insert a word count into a Word 2013 document, place the cursor where you would like the word count to appear (say in the Header or Footer) and then: 1 click the Insert tab. 2 click the Quick Parts icon (towards the right hand end of the toolbar). 3 on the drop down that appears, select Field...</code>	<code>0.0</code>
  <code>what is the difference between discipleship and evangelism</code>	<code>Discipleship, on the other hand, meant helping someone who was already a believer walk out the life of faith. The word “discipleship” brought to my mind a small group Bible study, a conversation across the table with another woman, or an accountability group. And I knew which one I preferred. As a result, the discipleship I offered others contained a lot of good information but lacked the transforming power that can only come from the gospel. (I was also, simply, a coward.). I am beginning to see that evangelism and discipleship are not all that different.</code>	<code>0.0</code>
  <code>what metal is a trophy made from</code>	<code>The trophy stands 36.5 centimetres (14.4 inches) tall and is made of 5 kg (11 lb) of 18 carat (75%) gold with a base (13 centimetres [5.1 inches] in diameter) containing t … wo layers of malachite. Making the world better, one answer at a time. Trophies can be made out of anything you want. however, aluminum is a very reliable and trustworthy metal and it.......... oh crap.......... i have to do a poo...</code>	<code>1.0</code>

• Loss: <code>BinaryCrossEntropyLoss</code> with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• remove_unused_columns: False
• load_best_model_at_end: True

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: False
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.0.0
• Transformers: 4.51.0
• PyTorch: 2.9.1+cu126
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.4-dev.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->