modernbert-embed-base_finetune_512

modernbert-embed-base trained on triplets

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base <!-- at revision d556a88e332558790b210f7bdbe87da2fa94a8d8 -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown -->

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Free-Law-Project/modernbert-embed-base_finetune_512")
# Run inference
sentences = [
    'search_document: This was eventually codified as part of G. L. c. 210, § 3, which also specified other grounds for dispensing with parental consent, such as current imprisonment of the parent for more than three years. Chapter 593, § 1, of the Acts of 1953, codified as G. L. c. 210, § 3A, first provided for an independent proceeding, prior to adoption proceedings proper, at which it could be determined whether parental consent was to be necessary for the adoption. Its purpose was to facilitate and expedite the process of adoption of children being held in temporary foster care. See the Department of Public Welfare recommendations, 1953 House Doc. No. 118, accompanying their draft bill,. 1953 House Doc. No. 124. The proceeding could be brought by the Department of Public Welfare or any appropriate child care agency having custody of the child. But the act was silent as to the standards to be applied in deciding when consent could be dispensed with, and in Consent to Adoption of a Minor, 345 Mass. 706 ( 1963 ), this court held that, in the absence of any other indication in the statute, the conditions set out in § 3 for direct adoptions were still to be met ; specifically, the court held that a finding of parental “ unsuitability, ” without a finding of * 638wilful desertion or neglect for a year, was not an adequate basis for a decree dispensing with the parental consent. The department had evidently not intended the § 3 conditions to be read into the independent § 3A proceeding. Therefore the department immediately sponsored St. 1964, c. 425, which provided that consent could be dispensed with “ if the court finds that the best interests of the child will be served by placement for adoption ” ; the court was not to be restricted by the § 3 conditions, but was to give “ due regard to the ability, capacity and fitness of the child ’ s parents. . . and to the plans proposed by the department or other agency initiating such petition. ” This statute thus broadened the factors the court could consider in deciding whether to proceed over the parent ’ s objections ; unsuitability besides desertion or neglect was now clearly an available ground.',
    'search_query: What are the legal standards for dispensing with parental consent in adoption cases?',
    'search_query: What are the tax implications of inheriting property from a deceased relative?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Triplet

• Dataset: dev
• Evaluated with <code>TripletEvaluator</code>

Triplet

• Dataset: dev
• Evaluated with <code>TripletEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Free-Law-Project/opinions-synthetic-query-512

• Size: 2,828 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 33 tokens</li><li>mean: 407.44 tokens</li><li>max: 487 tokens</li></ul>	<ul><li>min: 15 tokens</li><li>mean: 21.59 tokens</li><li>max: 34 tokens</li></ul>	<ul><li>min: 14 tokens</li><li>mean: 18.47 tokens</li><li>max: 27 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>search_document: DISTRICT COURT OF APPEAL OF THE STATE OF FLORIDA FOURTH DISTRICT EURICE McGILL, Appellant, v. STATE OF FLORIDA, Appellee. No. 4D17 - 1492 [ August 31, 2017 ] Appeal of order denying rule 3. 850 motion from the Circuit Court for the Seventeenth Judicial Circuit, Broward County ; Paul L. Backman, Judge ; L. T. Case No. 10 - 12523CF10A. Eurice McGill, Lake City, pro se. No appearance required for appellee. PER CURIAM. Affirmed. WARNER, DAMOORGIAN and KUNTZ, JJ., concur. * * * Not final until disposition of timely filed motion for rehearing.</code>	<code>search_query: What are the procedural outcomes of appealing a denied rule 3.850 motion in Florida?</code>	<code>search_query: What are the tax implications of forming an LLC in Florida?</code>
  <code>search_document: Twersky v Incorporated Vil. of Great Neck ( 2015 NY Slip Op 02755 ) Twersky v Incorporated Vil. of Great Neck 2015 NY Slip Op 02755 Decided on April 1, 2015 Appellate Division, Second Department Published by New York State Law Reporting Bureau pursuant to Judiciary Law § 431. This opinion is uncorrected and subject to revision before publication in the Official Reports. Decided on April 1, 2015 SUPREME COURT OF THE STATE OF NEW YORK Appellate Division, Second Judicial Department RANDALL T. ENG, P. J. LEONARD B. AUSTIN JEFFREY A. COHEN BETSY BARROS, JJ. 2014 - 07552 ( Index No. 9576 / 12 ) [ * 1 ] Sharon Twersky, respondent, v Incorporated Village of Great Neck, et al., defendants, FHM Mortgage Corp., et al., appellants. Cascone & Kluepfel, LLP, Garden City, N. Y. ( Howard B. Altman of counsel ), for appellants. Isaacson, Schiowitz & Korson, LLP, Rockville Centre, N. Y. ( Jeremy Schiowitz of counsel ), for respondent. DECISION & ORDER In an action to recover damages for...</code>	<code>search_query: What is the appellate court's role in reviewing motions for summary judgment in personal injury cases?</code>	<code>search_query: What are the tax implications of selling real estate in New York?</code>
  <code>search_document: ), entered June 17, 2014, as denied their motion for summary judgment dismissing the complaint and all cross claims insofar as asserted against them. ORDERED that the order is affirmed insofar as appealed from, with costs. On the evening of November 18, 2011, the plaintiff, while walking on a sidewalk abutting property then owned by the defendants FHM Mortgage Corp. and Killer B ' s Realty Holding Corp. ( hereinafter together the appellants ), allegedly slipped and fell on a driveway apron covered by a blanket of wet and slimy leaves. The plaintiff testified at her deposition that it was very dark in the area where the accident occurred and that the lamp posts in the vicinity did not provide much illumination. She also testified that the portion of the apron on which she slipped sloped down to meet the driveway. The appellants moved for summary judgment dismissing the complaint and all cross claims insofar as asserted against them. The Supreme Court denied their motion...</code>	<code>search_query: What is the legal responsibility of property owners for maintaining a safe environment on their premises?</code>	<code>search_query: What are the tax implications of selling real estate property for a profit?</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Free-Law-Project/opinions-synthetic-query-512

• Size: 489 evaluation samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 489 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 23 tokens</li><li>mean: 401.07 tokens</li><li>max: 482 tokens</li></ul>	<ul><li>min: 15 tokens</li><li>mean: 22.1 tokens</li><li>max: 35 tokens</li></ul>	<ul><li>min: 15 tokens</li><li>mean: 18.69 tokens</li><li>max: 26 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>search_document: Mr. Justice Mercur delivered the opinion of the court, November 20th 1882. Both parties claim title to this land under sheriff ’ s sale as the property of James Strouss. The defendant purchased at a sale made in December 1815, the plaintiff at one made in March 1880. The plaintiff seeks to impeach the validity of the first sale * 411on the ground that it was made in fraud of the creditors of Strouss. The law presumes that a public judicial sale is made in good faith. This presumption stands, unless overthrown by clear and satisfactory evidence of fraud or unfair means. The contention was one of fact. Much evidence Avas given bearing on the question, and some of it conflicting. The learned judge submitted the case to the jury in a clear and correct charge. He instructed them that if the sheriff ’ s sale was made with the intention of hindering, delaying or defeating creditors, and the purchaser had knowledge of such, it was null and void, although the full value of the ...</code>	<code>search_query: What constitutes fraud in a sheriff’s sale and how does it affect property titles?</code>	<code>search_query: What are the requirements for filing a patent application in the United States?</code>
  <code>search_document: We think the plaintiff has no reason to complain of this declaration of the law. No error is assigned thereto. Then, as to the application of the evidence tending to establish the fraud, the court affirmed a point of the plaintiff put in these words, “ under the plaintiff ’ s evidence tending to prove fraud on the part of the defendant, the jury will consider all the separate facts in evidence, whether each fact of itself would be sufficient or not to fasten fraud on her in the premises ; and they may consider separate facts, if they are connected by the evidence and tend to prove that the [ defendant entered into and carried out a scheme or plan, to purchase the land in dispute at an under value, and for the benefit of herself, and also for the benefit of James Strouss or his family. ” We do not deem it necessary to consider seriatim the twenty - five specifications of error. We do not think the article of agreement Avas prima facie fraudulent as to creditors ; nor do...</code>	<code>search_query: What legal principles govern the consideration of fraud in contracts involving property disputes?</code>	<code>search_query: What are the tax implications of selling inherited property in the United States?</code>
  <code>search_document: 217 N. J. Super. 541 ( 1987 ) 526 A. 2d 290 ALAN C. STAVER, PLAINTIFF, v. MARGARET STAVER, DEFENDANT. Superior Court of New Jersey, Chancery Division Bergen County, Family Part. March 11, 1987. * 543 Donald L. Garber for plaintiff ( Donald L. Garber, attorney ; Michael I. Lubin on the brief ). John Fiorello for defendant ( Feldman, Feldman, Hoffman & Fiorello, attorneys ). SIMON, MARGUERITE T., J. S. C. Plaintiff husband brings this motion seeking to terminate his obligation to pay alimony to defendant pursuant to a judgment of divorce entered September 6, 1974. Defendant wife brings a cross - motion for enforcement of the judgment. At the time of the entry of the final judgment, plaintiff was employed as an ordained minister earning approximately $ 12, 000 a year. The parties entered into a consensual agreement which was incorporated into the judgment. Two pertinent stipulations of the agreement are as follows : ( 1 ) " Said alimony of $ 500 per month shall continue i...</code>	<code>search_query: Can alimony obligations be modified or terminated based on retirement and financial changes?</code>	<code>search_query: What are the tax implications of inheriting property in New Jersey?</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->