soda-vec-negative-sampling

SODA-VEC Negative Sampling: Biomedical Sentence Embeddings

Model Overview

SODA-VEC Negative Sampling is a specialized sentence embedding model trained on 26.5M biomedical text pairs using the MultipleNegativesRankingLoss from sentence-transformers. This model is optimized for biomedical and life sciences applications, providing high-quality semantic representations for scientific literature.

Key Features

• 🧬 Biomedical Specialization: Trained exclusively on PubMed abstracts and titles
• 🔬 Large Scale: 26.5M training pairs from complete PubMed baseline (July 2024)
• ⚡ Modern Architecture: Based on ModernBERT-embed-base with 768-dimensional embeddings
• 🎯 Negative Sampling: Uses standard MultipleNegativesRankingLoss for robust contrastive learning
• 📊 Production Ready: Optimized training with FP16, gradient clipping, and cosine scheduling

Model Details

Base Model

• Architecture: ModernBERT-embed-base (nomic-ai/modernbert-embed-base)
• Embedding Dimension: 768
• Max Sequence Length: 768 tokens
• Parameters: ~110M

Training Configuration

• Loss Function: MultipleNegativesRankingLoss (sentence-transformers)
• Training Data: 26,473,900 biomedical text pairs
• Epochs: 3
• Effective Batch Size: 256 (32 per GPU × 4 GPUs × 2 gradient accumulation)
• Learning Rate: 1e-5 with cosine scheduling
• Optimization: AdamW with weight decay (0.01)
• Precision: FP16 for efficiency
• Hardware: 4x Tesla V100-DGXS-32GB

Dataset

Source Data

• Origin: Complete PubMed baseline (July 2024)
• Content: Scientific abstracts and titles from biomedical literature
• Quality: 99.7% retention after filtering (128-6,000 character abstracts)
• Splits: 99.6% train / 0.2% validation / 0.2% test

Data Processing

• Error pattern removal and quality filtering
• Balanced train/validation/test splits
• Character length filtering for optimal training
• Duplicate detection and removal

Performance & Use Cases

Intended Applications

• Literature Search: Semantic search across biomedical publications
• Research Discovery: Finding related papers and concepts
• Knowledge Mining: Extracting relationships from scientific text
• Document Classification: Categorizing biomedical documents
• Similarity Analysis: Comparing research abstracts and papers

Biomedical Domains

• Molecular Biology
• Clinical Medicine
• Pharmacology
• Genetics & Genomics
• Biochemistry
• Neuroscience
• Public Health

Usage

Installation

pip install sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('EMBO/soda-vec-negative-sampling')

# Encode biomedical texts
texts = [
    "CRISPR-Cas9 gene editing in human embryos",
    "mRNA vaccine efficacy against COVID-19 variants",
    "Protein folding mechanisms in neurodegenerative diseases"
]

embeddings = model.encode(texts)
print(f"Embeddings shape: {embeddings.shape}")  # (3, 768)

Semantic Search

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Query and corpus
query = "Alzheimer's disease biomarkers"
corpus = [
    "Tau protein aggregation in neurodegeneration",
    "COVID-19 vaccine development strategies", 
    "Beta-amyloid plaques in dementia patients"
]

# Encode
query_embedding = model.encode([query])
corpus_embeddings = model.encode(corpus)

# Find most similar
similarities = cosine_similarity(query_embedding, corpus_embeddings)[0]
best_match = np.argmax(similarities)
print(f"Best match: {corpus[best_match]} (similarity: {similarities[best_match]:.3f})")

Training Details

Loss Function

The model uses MultipleNegativesRankingLoss, which:

• Treats all other samples in a batch as negatives
• Optimizes for high similarity between related texts
• Provides robust contrastive learning without explicit negative sampling
• Well-established in sentence-transformers ecosystem

Training Process

• Duration: ~4 days on 4x V100 GPUs
• Steps: 310,239 total training steps
• Evaluation: Every 1000 steps (310 evaluations, 1.8% overhead)
• Monitoring: Real-time TensorBoard logging
• Checkpointing: Model saved at end of each epoch

Optimization Features

• Gradient clipping (max_norm=5.0) for training stability
• Weight decay regularization for generalization
• Cosine learning rate scheduling
• Loss-only evaluation for efficiency
• Reproducible training (seed=42)

Technical Specifications

Hardware Requirements

• Training: 4x Tesla V100-DGXS-32GB (recommended)
• Inference: Any GPU with 4GB+ VRAM, or CPU
• Memory: ~2GB GPU memory for inference

Software Dependencies

• sentence-transformers >= 2.0.0
• transformers >= 4.20.0
• torch >= 1.12.0
• Python >= 3.8

Comparison with SODA-VEC (VICReg)

Limitations

• Domain Specificity: Optimized for biomedical text, may not generalize to other domains
• Language: English-only training data
• Recency: Training data cutoff at July 2024
• Bias: May reflect biases present in PubMed literature

Citation

If you use this model in your research, please cite:

@misc{soda-vec-negative-sampling-2024,
  title={SODA-VEC Negative Sampling: Biomedical Sentence Embeddings},
  author={EMBO},
  year={2024},
  url={https://huggingface.co/EMBO/soda-vec-negative-sampling},
  note={Trained on 26.5M PubMed text pairs using MultipleNegativesRankingLoss}
}

License

This model is released under the same license as the base ModernBERT model. Please refer to the original model card for licensing details.

Acknowledgments

• Base Model: nomic-ai/modernbert-embed-base
• Training Framework: sentence-transformers
• Data Source: PubMed/MEDLINE database
• Infrastructure: EMBO computational resources

Model Card Contact

For questions about this model, please contact EMBO or open an issue in the associated repository.

---

Last Updated: August 2024
Model Version: 1.0
Training Completion: In Progress (ETA: 4 days)