ettin-encoder-17m-denseon-embeddistill

SentenceTransformer based on jhu-clsp/ettin-encoder-17m

This is a sentence-transformers model finetuned from jhu-clsp/ettin-encoder-17m on the msmarco-embed-dense_on dataset. It maps sentences & paragraphs to a 256-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: jhu-clsp/ettin-encoder-17m <!-- at revision 987607455c61e7a5bbc85f7758e0512ea6d0ae4c -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 256 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • msmarco-embed-dense_on

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 256, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'Healthcare is involved, directly or indirectly, with the provision of health services to individuals. These services can occur in a variety of work settings, including hospitals, clinics, dental offices, out-patient surgery centers, birthing centers, emergency medical care, home healthcare, and nursing homes. Healthcare workers face a number of serious safety and health hazards.',
]
documents = [
    'In a right triangle, the orthocenter is the polygon vertex of the right angle. When the vertices of a triangle are combined with its orthocenter, any one of the points is the orthocenter of the other three, as first noted by Carnot (Wells 1991). These four points therefore form an orthocentric system. The circumcenter and orthocenter are isogonal conjugates. The orthocenter lies on the Euler line.',
    'who makes fire extinguishers',
    'what is powdery mildew cannabis',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 256] [3, 256]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5302, 0.5367, 0.4435]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Dataset: NanoMSMARCO
• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

msmarco-embed-dense_on

• Dataset: msmarco-embed-dense_on
• Size: 973,530 training samples
• Columns: <code>text</code> and <code>label</code>
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	<ul><li>min: 4 tokens</li><li>mean: 45.82 tokens</li><li>max: 179 tokens</li></ul>	<ul><li>size: 768 elements</li></ul>

• Samples:

  text	label
  <code>how long background check bank returned</code>	<code>[-1.896484375, 0.288330078125, -0.406005859375, -1.6953125, 1.98046875, ...]</code>
  <code>Humira® (adalimumab) is a prescription medication licensed to treat certain inflammatory conditions that affect the joints, spine, or digestive system. Specific Humira uses include the treatment of: Ankylosing spondylitis. Crohn's disease. Juvenile idiopathic arthritis, also known as juvenile rheumatoid arthritis.</code>	<code>[-1.77734375, -0.072265625, -0.301513671875, -0.66748046875, 0.80810546875, ...]</code>
  <code>Bonfim is a municipality located in the mideast of the state of Roraima in Brazil. Its population is 12,626 and its area is 8,095 km². The city lies opposite the Takutu River from Lethem, Guyana. The Takutu River Bridge links Bonfim and Roraima with the town of Lethem and the Atlantic port of Georgetown, Guyana.</code>	<code>[-0.411865234375, 1.58984375, 0.76123046875, 1.6171875, 0.34375, ...]</code>

• Loss: <code>EmbedDistillLoss</code> with these parameters:

  {
      "distance_metric": "cosine",
      "projection_dim": 768
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• learning_rate: 0.0001
• warmup_steps: 0.1
• weight_decay: 0.01
• bf16: True
• load_best_model_at_end: True
• seed: 12
• dataloader_num_workers: 2

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 32
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 0.0001
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary>

• The bold row denotes the saved checkpoint.

</details>

Training Time

• Training: 37.8 minutes
• Evaluation: 12.9 seconds
• Total: 38.0 minutes

Framework Versions

• Python: 3.11.12
• Sentence Transformers: 5.6.1
• Transformers: 5.14.1
• PyTorch: 2.13.0+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

EmbedDistillLoss

@article{kim2023embeddistill,
    title={EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval},
    author={Kim, Seungyeon and Rawat, Ankit Singh and Zaheer, Manzil and Jayasumana, Sadeep and Sadhanala, Veeranjaneyulu and Jitkrittum, Wittawat and Menon, Aditya Krishna and Fergus, Rob and Kumar, Sanjiv},
    year={2023},
    eprint={2301.12005},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->