Downloads · 30 days
28
6% of all-time downloads
kokojake/modernbert-embed-base-fitness-health-matryoshka
modernbert-embed-base-fitness-health-matryoshka is a sentence similarity model from kokojake. Use it when you need a score for how close two texts are. It is set up for sentence-transformers. The card lists the license as apache-2.0.
This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual simi…
Downloads · 30 days
28
6% of all-time downloads
All-time downloads
443
Public
Parameters
149M
596 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors596 MB · 99%
From the Hugging Face model README
This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("kokojake/modernbert-embed-base-fitness-health-matryoshka")
# Run inference
sentences = [
'facilitate publication;\n•\u2009\x07Mobilise academic expertise for \ndeveloping training programmes and \nmobilising trainers.\n\t Weigh in on the debate around issues \nrelated to rehabilitation promotion and funding, promote best practices to \ninfluence policies that favour access \nto rehabilitation services and thereby \nmove toward advocacy actions.\n48\nUsers,\nDisabled people’s\norganisations\nService\nproviders\nDecision-makers User \ngroups\nLocal\nauthorities\nMinistry of \nHealth, Ministry \nof Social Action,\netc.\nUnited Nations \n(WHO, etc.)\nHospitals, \nReference\nrehabilitation centre\nProfessional \nassociations\nService provider groups\nTraining institutes\nCommunity- \nbased Services\nFederation\nand national\n associations\nHospital, \nHealth \ncare centres Network: actors that can be mobilised for physical \nand functional rehabilitation\nInternational\nNational\nLocal\nInstitutional donors\nFacilitation organisations* * \x07Organisations (IOs, NGOs, etc.), agencies, universities and research centres that facilitate the existence of physical \nand functional rehabilitation via national or international projects.\nInternational \n consortia (IDDC, etc.)\n International\n networks \n (CBR, WCPT, \n WFOT, ISPO,\nFATO, etc.)\nLevels of intervention © Handicap International, 2013\n \n \n49\n\xa0Intervention.\n\xa0modalities\u200a.\nThe Unit has technical resources specifically \npositioned to be able to reach the maximum',
'training programmes for rehabilitation professionals',
'risks of yo-yo dieting and heart disease',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
dim_768{
"truncate_dim": 768
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4789 |
| cosine_accuracy@3 | 0.4789 |
| cosine_accuracy@5 | 0.4789 |
| cosine_accuracy@10 | 0.5219 |
| cosine_precision@1 | 0.4789 |
| cosine_precision@3 | 0.4789 |
| cosine_precision@5 | 0.4789 |
| cosine_precision@10 | 0.4395 |
| cosine_recall@1 | 0.0601 |
| cosine_recall@3 | 0.1802 |
| cosine_recall@5 | 0.3003 |
| cosine_recall@10 | 0.5134 |
| cosine_ndcg@10 | 0.499 |
| cosine_mrr@10 | 0.4861 |
| cosine_map@100 | 0.5681 |
dim_512{
"truncate_dim": 512
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4742 |
| cosine_accuracy@3 | 0.4742 |
| cosine_accuracy@5 | 0.4742 |
| cosine_accuracy@10 | 0.5141 |
| cosine_precision@1 | 0.4742 |
| cosine_precision@3 | 0.4742 |
| cosine_precision@5 | 0.4742 |
| cosine_precision@10 | 0.4362 |
| cosine_recall@1 | 0.059 |
| cosine_recall@3 | 0.1769 |
| cosine_recall@5 | 0.2948 |
| cosine_recall@10 | 0.5078 |
| cosine_ndcg@10 | 0.4934 |
| cosine_mrr@10 | 0.4808 |
| cosine_map@100 | 0.5632 |
dim_256{
"truncate_dim": 256
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4555 |
| cosine_accuracy@3 | 0.4555 |
| cosine_accuracy@5 | 0.4555 |
| cosine_accuracy@10 | 0.4969 |
| cosine_precision@1 | 0.4555 |
| cosine_precision@3 | 0.4555 |
| cosine_precision@5 | 0.4555 |
| cosine_precision@10 | 0.4188 |
| cosine_recall@1 | 0.057 |
| cosine_recall@3 | 0.1711 |
| cosine_recall@5 | 0.2851 |
| cosine_recall@10 | 0.4882 |
| cosine_ndcg@10 | 0.4746 |
| cosine_mrr@10 | 0.4624 |
| cosine_map@100 | 0.5446 |
dim_128{
"truncate_dim": 128
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4352 |
| cosine_accuracy@3 | 0.4352 |
| cosine_accuracy@5 | 0.4352 |
| cosine_accuracy@10 | 0.4727 |
| cosine_precision@1 | 0.4352 |
| cosine_precision@3 | 0.4352 |
| cosine_precision@5 | 0.4352 |
| cosine_precision@10 | 0.3988 |
| cosine_recall@1 | 0.0545 |
| cosine_recall@3 | 0.1636 |
| cosine_recall@5 | 0.2726 |
| cosine_recall@10 | 0.4639 |
| cosine_ndcg@10 | 0.4522 |
| cosine_mrr@10 | 0.4414 |
| cosine_map@100 | 0.5208 |
dim_64{
"truncate_dim": 64
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.3945 |
| cosine_accuracy@3 | 0.3945 |
| cosine_accuracy@5 | 0.3945 |
| cosine_accuracy@10 | 0.4297 |
| cosine_precision@1 | 0.3945 |
| cosine_precision@3 | 0.3945 |
| cosine_precision@5 | 0.3945 |
| cosine_precision@10 | 0.3598 |
| cosine_recall@1 | 0.0499 |
| cosine_recall@3 | 0.1497 |
| cosine_recall@5 | 0.2495 |
| cosine_recall@10 | 0.4224 |
| cosine_ndcg@10 | 0.4109 |
| cosine_mrr@10 | 0.4004 |
| cosine_map@100 | 0.4763 |
| positive | anchor | |
|---|---|---|
| type | string | string |
| details | <ul><li>min: 7 tokens</li><li>mean: 239.56 tokens</li><li>max: 410 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 10.8 tokens</li><li>max: 26 tokens</li></ul> |
| positive | anchor |
|---|---|
| <code>values and preferences among older people <br>in relation to exercise, noting that older <br>people valued the outcomes of exercise <br>for maintaining health. They judged that <br>the evidence for older people was likely to be relevant to all adults and agreed <br>there was likely to be some uncertainty or <br>variability with respect to people’s values and <br>preferences for exercise and its outcomes. <br>Some GDG members suggested that given reasonably consistent benefit and very <br>little harms, there would be no important <br>uncertainty or variability regarding people’s <br>values on the outcomes of exercise. In the <br>absence of direct qualitative evidence, the GDG judged from their own experience <br>that resource requirements for structured <br>exercise programmes would vary by country <br>and setting, but in some settings might <br>be associated with moderate costs (for <br>structured exercise programmes, compared with self-managed physical activity). The GDG <br>noted that costs could also vary according to <br>the modality of ...</code> | <code>exercise preferences and outcomes variability among adults</code> |
| <code>ICRC, ICRC Hospital Design and Rehabilitation Guidelines, Vol. 1: Models Of Care, ICRC, Geneva, 2022: https://shop. icrc.org/icrc-hospital-design-and-rehabilitation-guidelines-volume-1-models-of-care-print-en.html</code> | <code>ICRC rehabilitation guidelines 2022</code> |
| <code>fitness training is guided by a health worker or (if feasible) performed self-directed by <br>the patient following education and advice.<br>Metacognitive <br>training<br>Metacognitive training aims to improve social functioning through reducing cognitive biases/psychotic symptoms (e.g. delusion, impaired self-awareness or insight). <br>Metacognitive training is usually provided as a structured group intervention during which participants perform exercises to reflect their own thinking and receive training in <br>strategies to cope with cognitive biases during daily routines. Metacognitive training is <br>guided by a health worker.<br>Mindfulness-<br>based approaches Mindfulness-based interventions aim to achieve a state of mindfulness in which a <br>person becomes more aware of their physical, mental, and emotional condition in the <br>present moment, without becoming judgemental. Mindfulness-based interventions (e.g. mindfulness-based cognitive therapy, acceptance and commitment therapy) <br>help people to pay attentio...</code> | <code>structured group interventions for metacognitive training</code> |
{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": -1
}
eval_strategy: epochper_device_train_batch_size: 32per_device_eval_batch_size: 16gradient_accumulation_steps: 16learning_rate: 2e-05num_train_epochs: 4lr_scheduler_type: cosinewarmup_ratio: 0.1bf16: Truetf32: Trueload_best_model_at_end: Trueoptim: adamw_torch_fusedbatch_sampler: no_duplicatesoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: epochprediction_loss_only: Trueper_device_train_batch_size: 32per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 16eval_accumulation_steps: Nonetorch_empty_cache_steps: Nonelearning_rate: 2e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 4max_steps: -1lr_scheduler_type: cosinelr_scheduler_kwargs: {}warmup_ratio: 0.1warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Truefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Truelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}tp_size: 0fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Nonehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseinclude_for_metrics: []eval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters:auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falseuse_liger_kernel: Falseeval_use_gather_object: Falseaverage_tokens_across_devices: Falseprompts: Nonebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportional| Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 |
|---|---|---|---|---|---|---|---|
| 0.4444 | 10 | 64.4729 | - | - | - | - | - |
| 0.8889 | 20 | 32.1029 | - | - | - | - | - |
| 1.0 | 23 | - | 0.4734 | 0.4741 | 0.4590 | 0.4271 | 0.3722 |
| 1.3111 | 30 | 23.9454 | - | - | - | - | - |
| 1.7556 | 40 | 19.7319 | - | - | - | - | - |
| 2.0 | 46 | - | 0.4934 | 0.4926 | 0.4723 | 0.4471 | 0.4021 |
| 2.1778 | 50 | 17.6381 | - | - | - | - | - |
| 2.6222 | 60 | 16.9329 | - | - | - | - | - |
| 3.0 | 69 | - | 0.498 | 0.4954 | 0.4746 | 0.4528 | 0.4089 |
| 3.0444 | 70 | 15.4096 | - | - | - | - | - |
| 3.4889 | 80 | 15.4012 | - | - | - | - | - |
| 3.8444 | 88 | - | 0.4990 | 0.4934 | 0.4746 | 0.4522 | 0.4109 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->