static-embedding-miriad

Static Embedding for MIRIAD Medical Retrieval

This is a sentence-transformers model trained on the retrieval and qa datasets. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer

<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->

• Maximum Sequence Length: inf tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • retrieval
  • qa
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): StaticEmbedding({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/static-embedding-miriad")
# Run inference
sentences = [
    'What instructions were given to patients regarding oral hygiene and care after the implant surgery?',
    "Patients were screened for study inclusion on the day of implant placement on a rolling admission basis. Medical history was reviewed prior to each implant procedure. Sociodemographic data were collected and included age, gender, race, and ethnicity. All implants were placed at posterior sites (premolar or molar) with the exception of one implant placed at an anterior (cuspid) location. A tooth was selected for evaluation as a control within the same quadrant as the implant site. The control tooth was the first available tooth mesial to the implant site and was separated from the implant by at least one natural tooth in order to avoid possible influence from the adjacent developing implant sulcus (34) .\n\n Probing depths, bleeding on probing, and supragingival plaque were recorded for the control tooth. Subgingival scaling was completed presurgically on the control tooth chosen to be sampled. During the implant surgery, flaps were reflected, and gingival thickness was measured from six different sites surrounding the implant with a Marquis probe (Hu-Friedy, Inc., Chicago, IL) (2). A transmucosal healing abutment was used for all patients. Implants were manufactured by Nobel Biocare (Yorba Linda, CA); root form, tapered implants were placed. Patients were instructed to avoid brushing at or near the surgical site for 1 week. All patients were prescribed and instructed to gently rinse with 0.12% chlorhexidine gluconate for 30 s twice per day for 1 week (2). Postoperative acetaminophen or hydrocodone with acetaminophen was advised for postoperative analgesia.\n\n Four-week GCF, PICF, and subgingival plaque sample collection. Medical history and exclusion criteria were reviewed for patients returning for the 4-week postoperative and sample collection visit. All patients were assessed for any signs of infection or failure of implant osseointegration at this appointment.\n\n Crevicular fluid samples, followed by subgingival plaque samples, were collected at the dental implant and control tooth sites on patients who continued to be eligible for study participation. Prior to crevicular fluid and subgingival plaque sample collection, supragingival plaque was removed around the implant abutment and control tooth by scaling. The implant and control tooth were isolated using cotton rolls and gently air dried. Crevicular fluid samples were then collected by inserting filter paper strips (Periopaper; Proflow, Amityville, NY), until slight resistance was felt, into the mesiobuccal and mesiolingual peri-implant and tooth sulci for 30 s. The two peri-implant crevicular fluid (PICF) samples for the implant were placed into a single microcentrifuge tube on ice beside the dentist's chair and frozen at Ϫ80°C within 10 min of collection. Likewise, the same procedure was followed for the two gingival crevicular fluid (GCF) samples taken from the control tooth.\n\n Peri-implant plaque samples were collected by inserting four separate sterile endodontic paper points (Henry Schein medium absorbent points; Melville, NY) into the mesial, distal, buccal, and lingual sites (i.e., circumferentially around the implant) for a period of 10 s each. Subgingival plaque samples collected from around the dental implant were pooled in a single microcentrifuge tube and frozen at Ϫ80°C within 10 min of collection. The same procedure was followed for the plaque samples taken from the control tooth. GCF, PICF, and subgingival plaque samples remained frozen at Ϫ80°C until analysis.\n\n Twelve-week GCF, PICF, and subgingival plaque sample collection. Twelve weeks after the initial implant placement, patients returned, and the same protocol was performed as described above for the 4-week sample collection appointment. All plaque, GCF, and PICF samples at both 4-week and 12-week visits were collected by the same investigator (P. G. Johnson).\n\n DNA extraction of plaque samples. DNA extraction was performed by the method of Martinez et al. (35) with slight modifications. Briefly, 200 l of ice-cold phosphate-buffered saline (PBS) was added to tubes containing the paper points and the tubes were vortexed for 15 min. Next, 750 l of lysis buffer and 20 mg/ml of lysozyme were added to the tubes, and the samples were then transferred into bead beating tubes containing 300 mg of disruption beads (0.1 mm; Research Products International Corp., Mount Prospect, IL). The samples were incubated for 15 min in a 37°C water bath.",
    '111 an earlier study, Boissonnault and Blaschak" found that only 11% of their subjects had rectus abdominis muscle separation below the umbilicus in their third trimester. The authors noted, however, that a higher incidence would have been found if the diastasis criteria had approached the normal linea alba width at this point rather than the 2 c m criterion suggested by -----+ I\n\n I(apandji3+ described the rectus abdomi~lis muscle as a powerful trunk flexor muscle operating by a lever system through the Iumbosacral and thoracolumbar joints. The rectus abdominis muscle\'s normal line of action is aligned vertically from the costal margin to the p u b i~.~.~~ Our results, however, show that by 30 weeks of gestation, the angle of insertion in the coronal and sagittal planes for rectus abdominis muscle had altered such that the muscle\'s line of action was deviated laterally and anteriorly, as shown in Figure 5 . A sinrplified force diagram for the rectus abdoniinis muscle at 30 weeks of gestation in the sagittal plane at the thoracolumbar and lumbosacral joints is show11 in Figure 6 . This diagram shows that the moment arm length, and therefore torque production of the rectus abdominis muscle about these joints in this plane, may be reduced at 30 weeks of gestation. The ability of the rectus abdominis muscle to flex the trunk is therefore possibly diminished. Whether the minimum change in angles of insertion at which reduced torque prodtrction of the rectus abdominis muscle will have a demonstrable affect on the muscle\'s functional capacity is unknown.\n\n -Umbilicus\n\n As we assumed that the adaptations of this representative abdominal muscle did not occur in isolation, we believe the ability to generate torque rnay be compromised across the entire muscle group. Hence, the second aim of this study was to examine the functional ability of the abdominal muscles, and the temporal relationship between functional ability and muscle adaptations, during pregnancy and into the postpartum period. Because there were few subjects in this study, which precluded the use of some statistical methods, we view this part of the study as a pilot investigation of abdominal muscle function during pregnancy and the postpartum period on a longitudinal basis.\n\n By 26 weeks of gestation, the ability to perform a curl-up type of abdominal exercise had diminished for all s u b jects. This decline continued to 38 weeks of gestation, at which time no subjects were successful in completi~lg a curl-up. Fast et a15 found that 22 out of 164 subjects at a mean of 38 weeks of gestation could successfully complete a hook-lying sit-up to a \\isually determined 40-degree angle from the horizontal. The higher percentage of successful subjects in the study by Fast e t ali lrldy be due to the use of a less extensive curl-up movement than that used in our study.\n\n We found the performance of curl-ups was improved postbirth. Five of the six subjects were successfill or moderately successful in performance of a curl-up at 8 weeks postbirth. This postbirth improvement supports the previous results of S p e n~e ,~ who found that 80% of her subjects could complete a similar curl-up exercise at 6 weeks postbirth.\n\n A comparison of the results for the two AMTs used in this study show that the AMT and curl-up results concur for test sessions during the pregnancy; however, these results are conflicting for the postbirth period. These apparent conflicts in AMT results also have been noted by Kendall and McCreary" for nongravid subjects. The AMT results showed that the ability to stabilize the pelvis against resistance while positioned supine generally remained compromised postbirth. In contrast, the abdonlinal muscles\' supine trunk flexion ability increased postbirth. The use of a curl-up as a f~lnctional test for the abdominal muscles during pregnancy, however, is questionable. During pregnancy, the uterus presents a physical obstruction to the close approximation of the thorax and pelvis, which is necessary to complete a curl-up. The ability to pelform the curl-up may be more related to the presence of this physical obstruction rather than to the functional ability of the muscles. In addition, supine trunk flexion may be assisted by the hip flexor~.:~5~"Thus, the performance of a curl-up in the postbirth period may not solely be a test of the abdominal muscles\' capabilities. In contrast, the MkfT is performed primarily by the abdominal muscles. 35 Potential assistance by the hip extensors acting to rotate the pelvis posteriorly is comprised due to the lack of a fixed distal attachment on the rotating limb.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5613, 0.0316],
#         [0.5613, 1.0000, 0.0152],
#         [0.0316, 0.0152, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Datasets: miriad-retrieval, miriad-qa, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with <code>InformationRetrievalEvaluator</code>

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with <code>NanoBEIREvaluator</code> with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Datasets

retrieval

• Dataset: retrieval at 596b9ab
• Size: 4,467,542 training samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 48 characters</li><li>mean: 106.6 characters</li><li>max: 317 characters</li></ul>	<ul><li>min: 2837 characters</li><li>mean: 4576.82 characters</li><li>max: 7387 characters</li></ul>

• Samples:

  anchor	positive
  <code>What factors may contribute to increased pulmonary conduit durability in patients who undergo the Ross operation compared to those with right ventricular outflow tract obstruction?<br></code>	<code>I n 1966, Ross and Somerville 1 reported the first use of an aortic homograft to establish right ventricle-to-pulmonary artery continuity in a patient with tetralogy of Fallot and pulmonary atresia. Since that time, pulmonary position homografts have been used in a variety of right-sided congenital heart lesions. Actuarial 5-year homograft survivals for cryopreserved homografts are reported to range between 55% and 94%, with the shortest durability noted in patients less than 2 years of age. 4 Pulmonary position homografts also are used to replace pulmonary autografts explanted to repair left-sided outflow disease (the Ross operation). Several factors may be likely to favor increased pulmonary conduit durability in Ross patients compared with those with right ventricular outflow tract obstruction, including later age at operation (allowing for larger homografts), more normal pulmonary artery architecture, absence of severe right ventricular hypertrophy, and more natural positioning of ...</code>
  <code>How does MCAM expression in hMSC affect the growth and maintenance of hematopoietic progenitors?</code>	<code>After culture in a 3-dimensional hydrogel-based matrix, which constitutes hypoxic conditions, MCAM expression is lost. Concordantly, Tormin et al. demonstrated that MCAM is down-regulated under hypoxic conditions. 10 Furthermore, it was shown by others and our group that oxygen tension causes selective modification of hematopoietic cell and mesenchymal stromal cell interactions in co-culture systems as well as influence HSPC metabolism. [44] [45] [46] Thus, the observed differences between Sharma et al. and our data in HSPC supporting capacity of hMSC are likely due to the different culture conditions used. Further studies are required to clarify the influence of hypoxia in our model system. Altogether these findings provide further evidence for the importance of MCAM in supporting HSPC. Furthermore, previous reports have shown that MCAM is down-regulated in MSC after several passages as well as during aging and differentiation. 19, 47 Interestingly, MCAM overexpression in hMSC enhance...</code>
  <code>What is the relationship between Fanconi anemia and breast and ovarian cancer susceptibility genes?<br></code>	<code>( 31 ) , of which 5% -10 % may be caused by genetic factors ( 32 ) , up to half a million of these patients may be at risk of secondary hereditary neoplasms. The historic observation of twofold to fi vefold increased risks of cancers of the ovary, thyroid, and connective tissue after breast cancer ( 33 ) presaged the later syndromic association of these tumors with inherited mutations of BRCA1, BRCA2, PTEN, and p53 ( 16 ) . By far the largest cumulative risk of a secondary cancer in BRCA mutation carriers is associated with cancer in the contralateral breast, which may reach a risk of 29.5% at 10 years ( 34 ) . The Breast Cancer Linkage Consortium ( 35 , 36 ) also documented threefold to fi vefold increased risks of subsequent cancers of prostate, pancreas, gallbladder, stomach, skin (melanoma), and uterus in BRCA2 mutation carriers and twofold increased risks of prostate and pancreas cancer in BRCA1 mutation carriers; these results are based largely on self-reported family history inf...</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

qa

• Dataset: qa at 596b9ab
• Size: 4,467,542 training samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 48 characters</li><li>mean: 106.6 characters</li><li>max: 317 characters</li></ul>	<ul><li>min: 116 characters</li><li>mean: 507.5 characters</li><li>max: 1009 characters</li></ul>

• Samples:

  anchor	positive
  <code>What factors may contribute to increased pulmonary conduit durability in patients who undergo the Ross operation compared to those with right ventricular outflow tract obstruction?<br></code>	<code>Several factors may contribute to increased pulmonary conduit durability in patients who undergo the Ross operation compared to those with right ventricular outflow tract obstruction. These factors include later age at operation (allowing for larger homografts), more normal pulmonary artery architecture, absence of severe right ventricular hypertrophy, and more natural positioning of the homograft. However, further systematic studies are needed to confirm these associations.</code>
  <code>How does MCAM expression in hMSC affect the growth and maintenance of hematopoietic progenitors?</code>	<code>MCAM expression in hMSC has been shown to support the growth of hematopoietic progenitors. It enhances the adhesion and migration of HSPC, potentially through direct cell-cell interactions. However, the putative interaction partner of MCAM on HSPC remains unknown. Additionally, MCAM expression in hMSC does not seem to regulate the expression or secretion of SDF-1, a key factor in HSPC homing and maintenance.</code>
  <code>What is the relationship between Fanconi anemia and breast and ovarian cancer susceptibility genes?<br></code>	<code>Fanconi anemia is a rare, autosomal recessive syndrome characterized by chromosomal instability, cancer susceptibility, and hypersensitivity to DNA cross-linking agents. It has been found that all known Fanconi anemia proteins cooperate with breast and/or ovarian cancer susceptibility gene products (BRCA1 and BRCA2) in a pathway required for cellular resistance to DNA cross-linking agents. This pathway, known as the "Fanconi anemia-BRCA pathway," is a DNA damage-activated signaling pathway that controls DNA repair. Methylation of one of the Fanconi anemia genes, FANCF, can lead to the inactivation of this pathway in breast and ovarian cancer, suggesting its importance in human carcinogenesis.</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Datasets

retrieval

• Dataset: retrieval at 596b9ab

• Size: 10,000 evaluation samples

• Columns: <code>anchor</code> and <code>positive</code>

• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 40 characters</li><li>mean: 106.65 characters</li><li>max: 227 characters</li></ul>	<ul><li>min: 2841 characters</li><li>mean: 4564.28 characters</li><li>max: 6653 characters</li></ul>

• Samples:

  anchor	positive
  <code>What are some hereditary cancer syndromes that can result in various forms of cancer?<br></code>	<code>Hereditary Cancer Syndromes, including Hereditary Breast and Ovarian Cancer (HBOC) and Lynch Syndrome (LS), can result in various forms of cancer due to germline mutations in cancer predisposition genes. While the major contributory genes for these syndromes have been identified and well-studied (BRCA1/ BRCA2 for HBOC and MSH2/MSH6/MLH1/PMS2/ EPCAM for LS), there remains a large percentage of associated cancer cases that are negative for germline mutations in these genes, including 80% of women with a personal or family history of breast cancer who are negative for BRCA1/2 mutations [1] . Similarly, between 30 and 50% of families fulfill stringent criteria for LS and test negative for germline mismatch repair gene mutations [2] . Adding complexity to these disorders is the significant overlap in the spectrum of cancers observed between various hereditary cancer syndromes, including many cancer susceptibility syndromes. Some that contribute to elevated breast cancer risk include Li-Frau...</code>
  <code>How do MAK-4 and MAK-5 exert their antioxidant properties?<br></code>	<code>Hybrid F1 mice were injected with urethane (300 mg/kg) at 8 days of age. A group was then put on a MAK-supplemented diet, another group was fed a standard pellet diet. At 36 weeks of age the mice were sacrificed and the livers examined for the presence of tumors mouse (Panel A) and for the number of nodules per mouse (Panel B) (* p < 0.05, ** P < 0.001). Statistical analysis was performed by Two Way ANOVA Test followed by Post Hoc Bonferroni analysis.

  We than measured the influence of the MAK-4+5 combination on the expression of the three liver-specific connexins (cx26, cx32, and cx43). The level of cx26 expression was similar in all the groups of mice treated with the MAK-supplemented diet and in the control (Figure 4, Panel A) . A significant, time-dependent increase in cx32 was observed in the liver of all the groups of MAK treated mice compared to the normal diet-fed controls. Cx32 expression increased 2-fold after 1 week of treatment, and 3-to 4-fold at 3 months (Figure 4, Pane...</code> | | <code>What are the primary indications for a decompressive craniectomy, and what role does neurocritical care play in determining the suitability of a patient for this procedure?</code> | <code>Decompressive craniectomy is a valid neurosurgical strategy now a day as an alternative to control an elevated intracranial pressure (ICP) and controlling the risk of uncal and/or subfalcine herniation, in refractory cases to the postural, ventilator, and pharmacological measures to control it. The neurocritical care and the ICP monitorization are key determinants to identify and postulate the inclusion criteria to consider a patient as candidate to this procedure, as it is always considered a rescue surgical technique. Head trauma and ischemic or hemorrhagic cerebrovascular disease with progressive deterioration due to mass effect are some of the cases that may require a decompressive craniectomy with its different variants. However, this procedure per se can have complications described in the postcraniectomy syndrome and may occur in short, medium, or even long term.

  1,2 The paradoxical herniation is a condition in which there is a deviation of the midline with mass effect, even t...</code> |

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

qa

• Dataset: qa at 596b9ab
• Size: 10,000 evaluation samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 40 characters</li><li>mean: 106.65 characters</li><li>max: 227 characters</li></ul>	<ul><li>min: 189 characters</li><li>mean: 492.49 characters</li><li>max: 915 characters</li></ul>

• Samples:

  anchor	positive
  <code>What are some hereditary cancer syndromes that can result in various forms of cancer?<br></code>	<code>Hereditary cancer syndromes, such as Hereditary Breast and Ovarian Cancer (HBOC) and Lynch Syndrome (LS), can result in various forms of cancer due to germline mutations in cancer predisposition genes. These syndromes are associated with an increased risk of developing specific types of cancer.</code>
  <code>How do MAK-4 and MAK-5 exert their antioxidant properties?<br></code>	<code>MAK-4 and MAK-5 have been shown to have antioxidant properties both in vitro and in vivo. These preparations contain multiple antioxidants such as alpha-tocopherol, beta-carotene, ascorbate, bioflavonoid, catechin, polyphenols, riboflavin, and tannic acid. These antioxidants are known to scavenge free radicals and reactive oxygen species (ROS) such as superoxide, hydroxyl, and peroxyl radicals, as well as hydrogen peroxide. In the present study, the antioxidant properties of MAK-4 and MAK-5 were confirmed in mice, with higher oxygen radical absorbance capacity (ORAC) values observed in mice fed the MAK-supplemented diet. Additionally, the activity of liver enzymes GPX, GST, and QR, which are involved in detoxification processes, were upregulated in the MAK-fed mice. This suggests that MAK-4 and MAK-5 may protect against carcinogenesis by reducing oxidative stress and enhancing detoxification processes.</code>
  <code>What are the primary indications for a decompressive craniectomy, and what role does neurocritical care play in determining the suitability of a patient for this procedure?</code>	<code>The primary indications for a decompressive craniectomy include refractory intracranial pressure (ICP) and progressive neurological deterioration due to mass effect from conditions like head trauma, or ischemic or hemorrhagic cerebrovascular disease. Neurocritical care and ICP monitoring are essential in identifying suitable candidates for the procedure, as it is considered a rescue surgical technique. These measures help to assess the patient's condition and determine the need for decompressive craniectomy in cases of elevated ICP.</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 2048
• num_train_epochs: 1
• learning_rate: 0.1
• lr_scheduler_type: cosine
• warmup_steps: 0.1
• bf16: True
• disable_tqdm: True
• project: static-embedding-miriad
• per_device_eval_batch_size: 2048
• push_to_hub: True
• hub_model_id: tomaarsen/static-embedding-miriad
• dataloader_num_workers: 4
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 2048
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 0.1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: True
• project: static-embedding-miriad
• trackio_space_id: trackio
• per_device_eval_batch_size: 2048
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: True
• hub_private_repo: None
• hub_model_id: tomaarsen/static-embedding-miriad
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary>

The model card is truncated. Read the rest on Hugging Face.