accphysbert_cased

AccPhysBERT

AccPhysBERT is a specialized sentence-embedding model fine-tuned for accelerator physics, capturing semantic nuances in this technical domain. It delivers state-of-the-art performance in tasks such as semantic search, citation classification, reviewer matching, and clustering of accelerator-physics literature.

---

Model Description

• Architecture: BERT-based, fine-tuned from PhysBERT (cased) using Supervised Contrastive Learning (SimCSE).
• Optimized For: Titles, abstracts, proposals, and full text from the accelerator-physics community.
• Notable Features:
  • Trained on 109 k accelerator-physics publications from INSPIRE HEP
  • Leverages 690 k citation pairs and 2 M synthetic query–source pairs
  • Trained via SentenceTransformers to produce dense, semantically rich embeddings

Developed by: Thorsten Hellert, João Montenegro, Marco Venturini, Andrea Pollastro
Funded by: US Department of Energy, Lawrence Berkeley National Laboratory
Model Type: Sentence embedding (BERT-based, SimCSE fine-tuned)
Language: English
License: CC BY 4.0
Paper: Domain-specific text embedding model for accelerator physics, Phys. Rev. Accel. Beams 28, 044601 (2025)
https://doi.org/10.1103/PhysRevAccelBeams.28.044601

---

Training Data

• Core Corpus:

  • 109,000 accelerator-physics publications (INSPIRE HEP category: "Accelerators")
  • Over 1 GB of full-text markdown-style text (via OCR/Nougat)

• Annotation Sources:

  • 690,000 citation pairs
  • 49 semantic categories labeled via ChatGPT-4o
  • 2,000,000 synthetic query–source pairs generated with LLaMA3-70B

---

Training Procedure

• Fine-tuning Method: SimCSE (contrastive loss)
• Hyperparameters:
  • Batch size: 512
  • Learning rate: 2e-4
  • Temperature: 0.05
  • Weight decay: 0.01
  • Optimizer: Adam
  • Epochs: 2
• Infrastructure: 32 × NVIDIA A100 GPUs @ NERSC
• Framework: SentenceTransformers

---

Evaluation Results

AccPhysBERT outperforms BERT, SciBERT, and large general-purpose embedding models in all accelerator-specific benchmarks.

---

Example Usage

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("thellert/accphysbert")
model = AutoModel.from_pretrained("thellert/accphysbert")

text = "We report on beam instabilities observed in the LCLS-II injector."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Use mean pooling (excluding [CLS] and [SEP])
token_embeddings = outputs.last_hidden_state[:, 1:-1, :]
sentence_embedding = token_embeddings.mean(dim=1)

---

Citation

If you use AccPhysBERT, please cite:

@article{Hellert_2025,
  title     = {Domain-specific text embedding model for accelerator physics},
  author    = {Hellert, Thorsten and Montenegro, João and Venturini, Marco and Pollastro, Andrea},
  journal   = {Physical Review Accelerators and Beams},
  volume    = {28},
  number    = {4},
  pages     = {044601},
  year      = {2025},
  publisher = {American Physical Society},
  doi       = {10.1103/PhysRevAccelBeams.28.044601},
  url       = {https://doi.org/10.1103/PhysRevAccelBeams.28.044601}
}

---

Contact

Thorsten Hellert
Lawrence Berkeley National Laboratory
📧 [email protected]

---

Acknowledgments

This model builds on PhysBERT and was trained using NERSC resources. Thanks to Alex Hexemer, Fernando Sannibale, and Antonin Sulc for their support and discussions.