custom-bge-baai-small-cfr-cms

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5 <!-- at revision 5c38ec7c405ec4b44b94cc5a9bb96e735b38267a -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Update to Pub. 100-04, Chapter 1 to Provide Language-Only Changes Updating ICD-10 and ASC Automation of the Request Claims Process Rescinded and replaced by Transmittal 3154 Adjustment Fiscal Intermediary Shared System Consistency Edit Implement National Uniform Billing Committee (NUBC) Revision Occurrence Span Code (OSC) Definition for Code Reporting the Service Location National Provider Identifier on Anti-Markup and Reference Laboratory Claims and replaced Transmittal the Reopening Process - Rescinded and replaced by Transmittal 3060',
    'Update to Pub. 100-04, Chapter 1 to Provide Language-Only Changes for Updating ICD-10 and ASC X12 Automation of the Request for Reopening Claims Process - Rescinded and replaced by Transmittal 3154 Adjustment to Fiscal Intermediary Shared System (FISS) Consistency Edit to Implement National Uniform Billing Committee (NUBC) Revision to Occurrence Span Code (OSC) Definition for Code 72 Reporting the Service Location National Provider Identifier (NPI) on Anti-Markup and Reference Laboratory Claims - Rescinded and replaced by Transmittal 3098 Automation of the Request for Reopening Claims Process - Rescinded and replaced by Transmittal 3060',
    'The issue is feasibility and not whether coverage is provided in one setting and not provided in another. For instance, an individual in need of daily skilled physical therapy might be able to receive the services needed on a more economical basis from an could not be made for the services because an expense limitation (if applicable) to the services of an independent physical therapist had been exceeded or because the patient was not enrolled in Part B, would not be a basis for determining that, as a practical',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9797, 0.0785],
#         [0.9797, 1.0000, 0.0807],
#         [0.0785, 0.0807, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 47,056 training samples
• Columns: <code>sentence_0</code> and <code>sentence_1</code>
• Approximate statistics based on the first 100 samples:

  	sentence_0	sentence_1
  type	string	string
  modality	text	text
  details	<ul><li>min: 26 tokens</li><li>mean: 86.86 tokens</li><li>max: 166 tokens</li></ul>	<ul><li>min: 44 tokens</li><li>mean: 125.96 tokens</li><li>max: 261 tokens</li></ul>

• Samples:

  sentence_0	sentence_1
  <code>For purposes of claims services submitted an (as § 410.79(b) this chapter), Medicare deems to assigned by the beneficiary (or the person request payment beneficiary's the assignment accepted the MDPP supplier.</code>	<code><> For purposes of claims for services submitted by an MDPP supplier (as defined at § 410.79(b) of this chapter), Medicare deems such claims to have been assigned by the beneficiary (or the person authorized to request payment on the beneficiary's behalf) and the assignment accepted by the MDPP supplier.</code>
  <code>() Claims primary and/or other services related of specific diseases or the or There has been by Secretary of in section 1899(c)(2)(B) Act is purposes of model. <> The ACO ACO and other entities performing functions services related to ACO activities are prohibited or gifts remuneration to Medicare beneficiaries for influencing Medicare beneficiary's designate or not designate ACO professional paragraph (e) this section. The ACO, ACO participants, ACO professionals, other or entities performing functions and services related to ACO activities must not, directly or commit any act or adopt policy that coerces or influences Medicare beneficiary's decision to or not ACO professional as for under paragraph (e) of not limited to the</code>	<code><> () Claims for primary care and/or other services related to treatment of one or more specific diseases or conditions targeted by the model; or <> () There has been a determination by the Secretary that waiver of the requirement in section 1899(c)(2)(B) of the Act is necessary solely for purposes of testing the model. <> The ACO, ACO participants, ACO providers/suppliers, ACO professionals, and other individuals or entities performing functions and services related to ACO activities are prohibited from providing or offering gifts or other remuneration to Medicare beneficiaries as inducements for influencing a Medicare beneficiary's decision to designate or not to designate an ACO professional under paragraph (e) of this section. The ACO, ACO participants, ACO providers/suppliers, ACO professionals, and other individuals or entities performing functions and services related to ACO activities must not, directly or indirectly, commit any act or omission, nor adopt any policy that coerce...</code>
  <code>d. If code 70 is entered, the type of bill must be 11X, 18X, 21X, or 41X. e. If code 71 is entered, the first digit of FL 4 must be "1," "2," or "4" and the f. If code 72 is entered, the type of bill must be 11X, 12X, 13X, 14X, 18X, 21X, 22X, 23X, 32X, 34X, 71X, 72X, 73X, 74X, 75X, 76X, 77X, 81X, 82X, or 85X. g. If code 74 is entered, the type of bill must be 11X, 13X, 14X, 18X, 21X, 34X,</code>	<code>d. If code 70 is entered, the type of bill must be 11X, 18X, 21X, or 41X. e. If code 71 is entered, the first digit of FL 4 must be "1," "2," or "4" and the f. If code 72 is entered, the type of bill must be 11X, 12X, 13X, 14X, 18X, 21X, 22X, 23X, 32X, 34X, 71X, 72X, 73X, 74X, 75X, 76X, 77X, 81X, 82X, or 85X. g. If code 74 is entered, the type of bill must be 11X, 13X, 14X, 18X, 21X, 34X,</code>

• Loss: <code>DenoisingAutoEncoderLoss</code> with these parameters:

  {
      "decoder_name_or_path": "BAAI/bge-small-en-v1.5",
      "need_retokenization": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Training Time

• Training: 1.5 hours

Framework Versions

• Python: 3.12.6
• Sentence Transformers: 5.6.0
• Transformers: 4.56.0
• PyTorch: 2.8.0+cu129
• Accelerate: 1.10.1
• Datasets: 5.0.0
• Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

DenoisingAutoEncoderLoss

@inproceedings{wang-2021-TSDAE,
    title = "TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning",
    author = "Wang, Kexin and Reimers, Nils and Gurevych, Iryna",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    pages = "671--688",
    url = "https://arxiv.org/abs/2104.06979",
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->