Downloads · 30 days
8
5% of all-time downloads
kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k
modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k is a sentence similarity model from kokojake. Use it when you need a score for how close two texts are. It is set up for sentence-transformers. The card lists the license as apache-2.0.
This is a sentence-transformers model finetuned from kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and ca…
Downloads · 30 days
8
5% of all-time downloads
All-time downloads
147
Public
Parameters
149M
596 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors596 MB · 99%
From the Hugging Face model README
This is a sentence-transformers model finetuned from kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k")
# Run inference
sentences = [
'Low back pain is \nthe leading cause of \ndisability globally across \nall ages and in both \nsexes, representing 8% \nof all YLDs in 2020 (10).',
'prevalence of low back pain by age and sex',
'BMI calculation in postpartum studies',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
dim_768{
"truncate_dim": 768
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.54 |
| cosine_accuracy@3 | 0.5435 |
| cosine_accuracy@5 | 0.5573 |
| cosine_accuracy@10 | 0.6335 |
| cosine_precision@1 | 0.54 |
| cosine_precision@3 | 0.5399 |
| cosine_precision@5 | 0.5347 |
| cosine_precision@10 | 0.4633 |
| cosine_recall@1 | 0.0375 |
| cosine_recall@3 | 0.1121 |
| cosine_recall@5 | 0.1837 |
| cosine_recall@10 | 0.307 |
| cosine_ndcg@10 | 0.4894 |
| cosine_mrr@10 | 0.5542 |
| cosine_map@100 | 0.3353 |
dim_512{
"truncate_dim": 512
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.5262 |
| cosine_accuracy@3 | 0.5292 |
| cosine_accuracy@5 | 0.5418 |
| cosine_accuracy@10 | 0.6248 |
| cosine_precision@1 | 0.5262 |
| cosine_precision@3 | 0.5259 |
| cosine_precision@5 | 0.52 |
| cosine_precision@10 | 0.4515 |
| cosine_recall@1 | 0.0366 |
| cosine_recall@3 | 0.1097 |
| cosine_recall@5 | 0.1793 |
| cosine_recall@10 | 0.3001 |
| cosine_ndcg@10 | 0.4769 |
| cosine_mrr@10 | 0.5405 |
| cosine_map@100 | 0.33 |
dim_256{
"truncate_dim": 256
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.5249 |
| cosine_accuracy@3 | 0.5283 |
| cosine_accuracy@5 | 0.5383 |
| cosine_accuracy@10 | 0.6248 |
| cosine_precision@1 | 0.5249 |
| cosine_precision@3 | 0.5247 |
| cosine_precision@5 | 0.5186 |
| cosine_precision@10 | 0.452 |
| cosine_recall@1 | 0.0365 |
| cosine_recall@3 | 0.1093 |
| cosine_recall@5 | 0.1786 |
| cosine_recall@10 | 0.3003 |
| cosine_ndcg@10 | 0.4768 |
| cosine_mrr@10 | 0.5393 |
| cosine_map@100 | 0.3291 |
dim_128{
"truncate_dim": 128
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4963 |
| cosine_accuracy@3 | 0.4989 |
| cosine_accuracy@5 | 0.5054 |
| cosine_accuracy@10 | 0.579 |
| cosine_precision@1 | 0.4963 |
| cosine_precision@3 | 0.4959 |
| cosine_precision@5 | 0.4886 |
| cosine_precision@10 | 0.4225 |
| cosine_recall@1 | 0.0344 |
| cosine_recall@3 | 0.103 |
| cosine_recall@5 | 0.1678 |
| cosine_recall@10 | 0.2805 |
| cosine_ndcg@10 | 0.4474 |
| cosine_mrr@10 | 0.5081 |
| cosine_map@100 | 0.3124 |
dim_64{
"truncate_dim": 64
}
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.4215 |
| cosine_accuracy@3 | 0.4236 |
| cosine_accuracy@5 | 0.4349 |
| cosine_accuracy@10 | 0.5145 |
| cosine_precision@1 | 0.4215 |
| cosine_precision@3 | 0.4209 |
| cosine_precision@5 | 0.4162 |
| cosine_precision@10 | 0.3711 |
| cosine_recall@1 | 0.0291 |
| cosine_recall@3 | 0.0871 |
| cosine_recall@5 | 0.1422 |
| cosine_recall@10 | 0.2458 |
| cosine_ndcg@10 | 0.3886 |
| cosine_mrr@10 | 0.4353 |
| cosine_map@100 | 0.2758 |
| positive | anchor | |
|---|---|---|
| type | string | string |
| details | <ul><li>min: 11 tokens</li><li>mean: 229.18 tokens</li><li>max: 412 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 11.11 tokens</li><li>max: 45 tokens</li></ul> |
| positive | anchor |
|---|---|
| <code>A total of 5,697 postmenopausal women were included in the<br>meta-analysis. The mean age of participants was ranged from 51 to ~89 yrs., and the mean BMI was ranged from 21 to 34 kg.m2.<br>Sample size of individual studies was ranged from 14 to 320<br>participants. To increase the generalizability of our meta-analysis results, postmenopausal women regardless of their health status,<br>comprised a wide range of health (absence of disease) and chronic<br>disease characteristics (metabolic diseases, cardiovascular diseases, cancer, and osteoporosis) were included. Full details of participant<br>characteristics are summarized in Supplementary Table 1.<br>Intervention characteristics<br>Exercise training characteristics are summarized in Supplementary Table 1. All included studies compared the effects of exercise training<br>versus a control group using random allocation. Intervention durations<br>of included studies was ranged from 4 weeks to 18 months, while frequency of exercise sessions was ranged from 1 to 7 per w...</code> | <code>effects of exercise training on postmenopausal women with chronic diseases</code> |
| <code>inform care planning, including the need for a referral or follow-up. <br>Assessment <br>of nutritional <br>status<br>Nutritional status describes the state of the body in relation to the consumption and utilization of nutrients, and can be classified as well-nourished or malnourished (under- <br>or over-nourished). The assessment of nutritional status uses anthropometric measures to assess body composition (measurement of weight, height, body mass index, body <br>circumferences and skinfold thickness), laboratory tests to assess biochemical parameters, clinical assessment of comorbid conditions, and interviewing to assess dietary practices. <br>Assessment aims to ascertain the impact of the nutritional status on health and functioning, <br>and inform care planning, including the need for referral or follow-up. Assessment of <br>oedema<br>Oedema (e.g. peripheral or lymphoedema) describes an abnormal fluid volume in the circulatory system or in the interstitial space. The assessment of oedema (including <br>initial scr...</code> | <code>nutritional status impact on health and functioning</code> |
| <code>required<br> • Patient sitting or lying <br>(if under anaesthesia)<br> • Tubular bandage<br>(if needed)<br>Ø 10 cm<br> • Padding bandage<br>1 roll<br> • POP 3 rolls of 15 cm<br> • Elastic bandage<br>2 rolls of 15 cm, <br>1 roll of 10 cm<br> • Adhesive tape<br>2.5 cm<br> • Triangular <br>bandage Edge<br>A. Senet/ICRC<br>Table 3.7: Long arm slabs at a glance<br>Method of application<br>Refer to the general procedure for slabs (p. 45) for the first steps. Mark the proximal and distal landmarks.<br>P. Ley/ICRC<br> SLABS<br>77<br>Prepare six to eight layers of POP bandages of the required <br>length. Place the wet slab on the limb and mould it.<br>Secure the slab with elastic bandages.<br>P. Ley/ICRC<br>P. Ley/ICRC<br>P. Ley/ICRC<br>78 PLASTER OF PARIS AND OTHER FRACTURE IMMOBILIZATION METHODS<br>When the POP slab is bent around the elbow, pay special <br>attention to avoid wrinkles, which can cause pain. Make sure the ulnar nerve is not compressed by asking <br>the patient if the inside of their elbow is comfortable.<br>After bandaging, maintain the elbow and wrist in the proper <br>posi...</code> | <code>long arm slab application procedure</code> |
{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": -1
}
eval_strategy: epochper_device_train_batch_size: 32per_device_eval_batch_size: 16gradient_accumulation_steps: 16learning_rate: 2e-05lr_scheduler_type: cosinewarmup_ratio: 0.1bf16: Truetf32: Trueload_best_model_at_end: Trueoptim: adamw_torch_fusedbatch_sampler: no_duplicatesoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: epochprediction_loss_only: Trueper_device_train_batch_size: 32per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 16eval_accumulation_steps: Nonetorch_empty_cache_steps: Nonelearning_rate: 2e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 3max_steps: -1lr_scheduler_type: cosinelr_scheduler_kwargs: {}warmup_ratio: 0.1warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Truefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Truelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}tp_size: 0fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Nonehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseinclude_for_metrics: []eval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters:auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falseuse_liger_kernel: Falseeval_use_gather_object: Falseaverage_tokens_across_devices: Falseprompts: Nonebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportional| Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 |
|---|---|---|---|---|---|---|---|
| 0.2462 | 10 | 13.2182 | - | - | - | - | - |
| 0.4923 | 20 | 12.361 | - | - | - | - | - |
| 0.7385 | 30 | 10.9108 | - | - | - | - | - |
| 0.9846 | 40 | 10.1159 | 0.4810 | 0.4740 | 0.4704 | 0.4424 | 0.3798 |
| 1.2462 | 50 | 9.145 | - | - | - | - | - |
| 1.4923 | 60 | 7.7837 | - | - | - | - | - |
| 1.7385 | 70 | 7.6298 | - | - | - | - | - |
| 1.9846 | 80 | 7.9102 | 0.4889 | 0.4786 | 0.4790 | 0.4453 | 0.3867 |
| 2.2462 | 90 | 7.5969 | - | - | - | - | - |
| 2.4923 | 100 | 6.8696 | - | - | - | - | - |
| 2.7385 | 110 | 7.2096 | - | - | - | - | - |
| 2.9846 | 120 | 7.2675 | 0.4894 | 0.4769 | 0.4768 | 0.4474 | 0.3886 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->