us-openfda-drug-embed-v1

SentenceTransformer based on FremyCompany/BioLORD-2023

U.S. openFDA Drug Product Research Model

This experimental model was fine-tuned on structured drug-product records from the official openFDA National Drug Code Directory. Training text includes brand name, active ingredients, strength, dosage form, and route.

The openFDA source data is generally available under CC0/public-domain terms; the base model and software dependencies retain their own terms. This model is for research and catalog-matching experiments. It must not be used for diagnosis, prescribing, dosage decisions, interaction checks, or automatic medicine substitution.

Held-out evaluation:

• nearest-neighbour exact-product accuracy: 0.6585 -> 0.8171
• validation triplet accuracy: 0.8995 -> 0.9416
• normalized separation margin: 2.5155 -> 2.8190

This is a sentence-transformers model finetuned from FremyCompany/BioLORD-2023 on the pairs and triplets datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: FremyCompany/BioLORD-2023 <!-- at revision 167aab527b238a50ca65224e6319215d2ff4fc9f -->
• Maximum Sequence Length: 32 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • pairs
  • triplets

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MPNetModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Brand: smart care island mango hand sanitizer. Ingredients: alcohol. Strength: 70 ml/100ml. Form: spray. Route: topical.',
    'Brand: ashleybelle moisturizing hand sanitizer cranberry sage. Ingredients: alcohol. Strength: 70 ml/100ml. Form: spray. Route: topical.',
    'Brand: walgreens dandruff itchy dry scalp defense anti-dandruff. Ingredients: pyrithione zinc. Strength: 10 mg/ml. Form: shampoo. Route: topical.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8526, 0.4529],
#         [0.8526, 1.0000, 0.4206],
#         [0.4529, 0.4206, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Composition

• Dataset: comp
• Evaluated with <code>main.CompositionEvaluator</code>

Triplet

• Dataset: val
• Evaluated with <code>TripletEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Datasets

pairs

• Dataset: pairs
• Size: 13,331 training samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 27 tokens</li><li>mean: 31.86 tokens</li><li>max: 32 tokens</li></ul>	<ul><li>min: 28 tokens</li><li>mean: 31.86 tokens</li><li>max: 32 tokens</li></ul>

• Samples:

  anchor	positive
  <code>Brand: gps topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: primo topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>
  <code>Brand: bencocaine topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: tiger supply inc topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>
  <code>Brand: safco sensicaine ultra topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: advance topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {
      "guide": "SentenceTransformer(None)",
      "temperature": 0.01,
      "mini_batch_size": 64,
      "mini_batch_num_tokens": null,
      "margin_strategy": "absolute",
      "margin": 0.0,
      "contrast_anchors": true,
      "contrast_positives": true,
      "gather_across_devices": false
  }
  

triplets

• Dataset: triplets
• Size: 27,325 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	<ul><li>min: 27 tokens</li><li>mean: 31.8 tokens</li><li>max: 32 tokens</li></ul>	<ul><li>min: 28 tokens</li><li>mean: 31.88 tokens</li><li>max: 32 tokens</li></ul>	<ul><li>min: 28 tokens</li><li>mean: 31.61 tokens</li><li>max: 32 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>Brand: candee caine topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: advance topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: 7 select oral pain maximum strength relief. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: topical.</code>
  <code>Brand: kolorz topical anesthetic blue rasberry. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: kolorz topical anesthetic triple mint. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: lubelife climax control delay. Ingredients: benzocaine. Strength: 7.5 g/100ml. Form: spray. Route: topical.</code>
  <code>Brand: quala topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: kolorz topical anesthetic cotton candy. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.</code>	<code>Brand: hurricaine topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental | periodontal.</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {
      "guide": "SentenceTransformer(None)",
      "temperature": 0.01,
      "mini_batch_size": 64,
      "mini_batch_num_tokens": null,
      "margin_strategy": "absolute",
      "margin": 0.0,
      "contrast_anchors": true,
      "contrast_positives": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 512
• num_train_epochs: 1.0
• learning_rate: 1e-05
• warmup_steps: 0.1
• bf16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 512
• num_train_epochs: 1.0
• max_steps: -1
• learning_rate: 1e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.9 minutes
• Evaluation: 0.8 seconds
• Total: 1.9 minutes

Framework Versions

• Python: 3.12.10
• Sentence Transformers: 5.7.0
• Transformers: 5.14.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->