Downloads · 30 days
6
4% of all-time downloads
kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs
modernbert-embed-base-fitness-health-matryoshka-8-epochs is a sentence similarity model from kokojake. Use it when you need a score for how close two texts are. It is set up for sentence-transformers. The card lists the license as apache-2.0.
This is a sentence-transformers model finetuned from kokojake/modernbert-embed-base-fitness-health-matryoshka on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used…
Downloads · 30 days
6
4% of all-time downloads
All-time downloads
140
Public
Parameters
149M
596 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors596 MB · 99%
From the Hugging Face model README
This is a sentence-transformers model finetuned from kokojake/modernbert-embed-base-fitness-health-matryoshka on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs")
# Run inference
sentences = [
'With this policy paper we offer our teams and partners the basis for understanding what we \nbelieve physical and functional rehabilitation covers today. While we can look back and see how far we have come, we are also aware of the changes we have yet to go through. Hence it \nhighlights the directions we will be taking in the coming years, for example the user-centred approach, the quality and sustainability of services in developing countries, improving the \nprofessional training process, and connecting with user groups.\nWe have a small, highly-motivated and ambitious team that, with limited resources (always too limited!), manages daily tours-de-force to improve our practices, capitalise, train, \ninnovate, structure and improve the position of physical and functional rehabilitation in the world. This document follows naturally from that, highlighting how the medical field plays an \nessential role in enhancing the social participation of people with disabilities. The beneficiary becomes a new participant in his own health, allowing him to be a stakeholder in building the \nsocieties of today and tomorrow.\nThanks to all who contributed, and happy reading.',
'role of medical field in social participation for disabilities',
'selection and adjustment of assistive products for patient independence',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
dim_768{
"truncate_dim": 768
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.5211 |
| cosine_accuracy@3 | 0.5211 |
| cosine_accuracy@5 | 0.5211 |
| cosine_accuracy@10 | 0.5578 |
| cosine_precision@1 | 0.5211 |
| cosine_precision@3 | 0.5211 |
| cosine_precision@5 | 0.5211 |
| cosine_precision@10 | 0.4754 |
| cosine_recall@1 | 0.0649 |
| cosine_recall@3 | 0.1947 |
| cosine_recall@5 | 0.3245 |
| cosine_recall@10 | 0.5487 |
| cosine_ndcg@10 | 0.5375 |
| cosine_mrr@10 | 0.5272 |
| cosine_map@100 | 0.606 |
dim_512{
"truncate_dim": 512
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.5062 |
| cosine_accuracy@3 | 0.5062 |
| cosine_accuracy@5 | 0.5062 |
| cosine_accuracy@10 | 0.5461 |
| cosine_precision@1 | 0.5062 |
| cosine_precision@3 | 0.5057 |
| cosine_precision@5 | 0.5056 |
| cosine_precision@10 | 0.4611 |
| cosine_recall@1 | 0.0636 |
| cosine_recall@3 | 0.1904 |
| cosine_recall@5 | 0.3173 |
| cosine_recall@10 | 0.5348 |
| cosine_ndcg@10 | 0.5233 |
| cosine_mrr@10 | 0.5129 |
| cosine_map@100 | 0.595 |
dim_256{
"truncate_dim": 256
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4859 |
| cosine_accuracy@3 | 0.4859 |
| cosine_accuracy@5 | 0.4859 |
| cosine_accuracy@10 | 0.5227 |
| cosine_precision@1 | 0.4859 |
| cosine_precision@3 | 0.4859 |
| cosine_precision@5 | 0.4859 |
| cosine_precision@10 | 0.4407 |
| cosine_recall@1 | 0.0612 |
| cosine_recall@3 | 0.1837 |
| cosine_recall@5 | 0.3062 |
| cosine_recall@10 | 0.5121 |
| cosine_ndcg@10 | 0.5017 |
| cosine_mrr@10 | 0.4921 |
| cosine_map@100 | 0.5785 |
dim_128{
"truncate_dim": 128
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4688 |
| cosine_accuracy@3 | 0.4688 |
| cosine_accuracy@5 | 0.4688 |
| cosine_accuracy@10 | 0.507 |
| cosine_precision@1 | 0.4688 |
| cosine_precision@3 | 0.4688 |
| cosine_precision@5 | 0.4686 |
| cosine_precision@10 | 0.43 |
| cosine_recall@1 | 0.0584 |
| cosine_recall@3 | 0.1753 |
| cosine_recall@5 | 0.292 |
| cosine_recall@10 | 0.4963 |
| cosine_ndcg@10 | 0.4854 |
| cosine_mrr@10 | 0.4751 |
| cosine_map@100 | 0.5618 |
dim_64{
"truncate_dim": 64
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4141 |
| cosine_accuracy@3 | 0.4141 |
| cosine_accuracy@5 | 0.4141 |
| cosine_accuracy@10 | 0.4586 |
| cosine_precision@1 | 0.4141 |
| cosine_precision@3 | 0.4141 |
| cosine_precision@5 | 0.4141 |
| cosine_precision@10 | 0.3875 |
| cosine_recall@1 | 0.0512 |
| cosine_recall@3 | 0.1535 |
| cosine_recall@5 | 0.2558 |
| cosine_recall@10 | 0.4471 |
| cosine_ndcg@10 | 0.4339 |
| cosine_mrr@10 | 0.4215 |
| cosine_map@100 | 0.5068 |
| positive | anchor | |
|---|---|---|
| type | string | string |
| details | <ul><li>min: 10 tokens</li><li>mean: 244.14 tokens</li><li>max: 412 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 10.92 tokens</li><li>max: 39 tokens</li></ul> |
| positive | anchor |
|---|---|
| <code>certainty as rated in the Cochrane <br>review (89) due to indirectness], 35 <br>trials); and<br> ›<br> improve function (trivial effect, low certainty evidence [downgraded <br>#<br>Review findings: values and <br>preferences relevant to older <br>people<br>GRADE-<br>CERQual <br>assessment <br>of confidence<br>Explanation of confidence <br>assessment<br>12 Older people emphasized the <br>importance of continuity of <br>physical exercises to maintain <br>mobility and reduce pain. A lack of <br>continuity of physical exercise and <br>instruction could have adverse <br>effects.<br>LOW No/very minor concerns <br>regarding methodological <br>limitations, moderate <br>concerns regarding coherence, <br>minor concerns regarding <br>adequacy, and minor concerns <br>regarding relevance.<br>13 Older people also valued <br>educational materials to <br>accompany exercise programmes, <br>such as drawings and descriptions <br>of the exercises.<br>LOW<br>Minor concerns regarding <br>methodological limitations, no/very minor concerns <br>regarding coherence, serious <br>concerns regarding adequacy, <br>and serious con...</code> | <code>importance of physical exercise continuity for older adults mobility and pain reduction</code> |
| <code>Phosphodiesterase-5 <br>inhibitors<br>Prescription and/or administration (if injection) of the medicine and providing <br>education and advice on the safe intake or administration (if self-directed) and potential adverse effects of the medicine.<br>Physical exercise <br>training<br>A variety of physical exercises (e.g. aerobic or strengthening exercises, balance or <br>coordination exercises, mind–body exercises), with or without weight-bearing, are suitable to improve exercise capacity, muscle strength, joint mobility, voluntary <br>movement, balance, gait and walking, as well as helping to reduce pain and fatigue. Regular physical exercise training (including education and advice on exercises) <br>is planned according to an individual’s needs, guided or assisted and, if feasible, <br>performed self-directed following education and advice on the appropriate exercises.</code> | <code>Phosphodiesterase-5 inhibitors prescription administration and adverse effects</code> |
| <code>and occasional guidelines (Bach-Faig 2011): <br>Every day <br>• <br>Three main meals should contain three basic elements: o Cereals: 1-2 servings per meal (preferably whole grain), such as bread, pasta, <br>rice, and couscous</code> | <code>three main meals basic elements cereals whole grain</code> |
{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": -1
}
eval_strategy: epochper_device_train_batch_size: 32per_device_eval_batch_size: 16gradient_accumulation_steps: 16learning_rate: 2e-05num_train_epochs: 4lr_scheduler_type: cosinewarmup_ratio: 0.1bf16: Truetf32: Trueload_best_model_at_end: Trueoptim: adamw_torch_fusedbatch_sampler: no_duplicatesoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: epochprediction_loss_only: Trueper_device_train_batch_size: 32per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 16eval_accumulation_steps: Nonetorch_empty_cache_steps: Nonelearning_rate: 2e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 4max_steps: -1lr_scheduler_type: cosinelr_scheduler_kwargs: {}warmup_ratio: 0.1warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Truefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Truelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}tp_size: 0fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Nonehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseinclude_for_metrics: []eval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters:auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falseuse_liger_kernel: Falseeval_use_gather_object: Falseaverage_tokens_across_devices: Falseprompts: Nonebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportional| Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 |
|---|---|---|---|---|---|---|---|
| 0.4444 | 10 | 16.2146 | - | - | - | - | - |
| 0.8889 | 20 | 15.6805 | - | - | - | - | - |
| 1.0 | 23 | - | 0.5287 | 0.5234 | 0.5034 | 0.4812 | 0.4255 |
| 1.3111 | 30 | 13.2495 | - | - | - | - | - |
| 1.7556 | 40 | 13.4064 | - | - | - | - | - |
| 2.0 | 46 | - | 0.5320 | 0.5199 | 0.5019 | 0.4776 | 0.4306 |
| 2.1778 | 50 | 11.3483 | - | - | - | - | - |
| 2.6222 | 60 | 11.7323 | - | - | - | - | - |
| 3.0 | 69 | - | 0.5375 | 0.5216 | 0.5014 | 0.4816 | 0.4288 |
| 3.0444 | 70 | 11.3371 | - | - | - | - | - |
| 3.4889 | 80 | 10.2106 | - | - | - | - | - |
| 3.8444 | 88 | - | 0.5375 | 0.5233 | 0.5017 | 0.4854 | 0.4339 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->