ALIA-MrBERT-es-snomed-dental-ner-multiclass-ctx8192

ALIA MrBERT-es Snomed Dental Multiclass NER Model (Context 8192)

This repository contains ALIA-MrBERT-es-snomed-dental-ner-multiclass-ctx8192, a Spanish dentistry domain multiclass Named Entity Recognition (NER) model. It is built upon MrBERT-es, a bilingual (Spanish-English) foundational language model based on the ModernBERT architecture, and fine-tuned on dentistry-specific clinical data.

Unlike binary or single-class entity extraction models, this model is a direct multiclass token classifier. It detects dental clinical entities and assigns them directly to their corresponding SNOMED CT codes in a single inference step, utilizing a BIO (Beginning, Inside, Outside) tagging schema tailored for each concept.

[!WARNING] DISCLAIMER: This model is a domain-specific proof-of-concept designed to demonstrate multiclass entity recognition and direct coding in the Spanish dentistry domain. Clinical decisions should always be verified by qualified dental professionals and clinical experts. The model should not be used as a standalone diagnostic tool.

---

Model Details

Model Lineage

ModernBERT (architecture)
       ↓
  MrBERT-es (BSC-LT)
  Bilingual ES/EN encoder
  150M parameters
       ↓
  ALIA-MrBERT-es-snomed-dental-ner-multiclass-ctx8192 (UJA)
  Dentistry Multiclass Named Entity Recognition fine-tuning
  ~400 SNOMED CT clinical codes (Direct Coding)

Key Features

• 🦷 Domain: Spanish dental and dentistry clinical reports.
• 📐 Architecture: ModernBERT adapted for Multiclass Token Classification (NER).
• 📏 Long context: Up to 8,192 tokens, enabling complete clinical histories or multi-turn dental dialogues to be processed in a single pass.
• 🏷️ Direct SNOMED CT Coding: Classifies tokens directly into specific SNOMED CT concepts, bypassing the need for a separate downstream entity linking step.
• ⚙️ BIO Schema: Assigns B-{SNOMED_CODE} and I-{SNOMED_CODE} tags to extract exact entity boundaries and classifications simultaneously.

Architecture

This model utilizes the ModernBERT architecture, extended with a multiclass token classification head:

---

Training

Training Script & Strategy

The model was fine-tuned using the training pipeline defined in train_multiclass.py. The strategy maps spans in the dental corpus to BIO-formatted labels of the form B-{SNOMED_CODE} and I-{SNOMED_CODE}, where each code corresponds to one of the dental clinical terms.

• Loss Function: Cross-Entropy Loss at the token level (ignoring padding tokens with a label of -100).
• Evaluation Metric: Strict Token Micro-F1, which calculates True Positives, False Positives, and False Negatives globally. If the model predicts an entity with an incorrect SNOMED CT code, it is penalized as both a False Positive for the predicted class and a False Negative for the gold class.

Training Hyperparameters

The model was trained using the following parameters:

---

Intended Use

Direct Use

• End-to-end Dental Entity Extraction and Coding: Identifying and mapping dental pathologies, anatomy, and procedures directly to SNOMED CT codes in a single step.
• Clinical Text Structuring: Structuring unstructured dental patient files into standardized SNOMED CT term representations.

Out-of-Scope Use

• General clinical Named Entity Recognition outside of dentistry.
• Non-Spanish clinical texts.
• Text-generation tasks (this is an encoder-only model).
• Automated diagnostic or treatment decisions without human expert validation.

---

How to Use

With HuggingFace pipeline

from transformers import pipeline

# Load the multiclass token classification pipeline
ner_pipeline = pipeline(
    "token-classification",
    model="SINAI/ALIA-MrBERT-es-snomed-dental-ner-multiclass-ctx8192",
    aggregation_strategy="simple"
)

# Clinical text example
clinical_text = "El paciente presenta caries dental en el canino inferior derecho y apiñamiento severo."

# Run inference
entities = ner_pipeline(clinical_text)
for entity in entities:
    print(f"Texto: {entity['word']} | Código SNOMED: {entity['entity_group']} | Confianza: {entity['score']:.4f}")

With transformers (Manual Inference)

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = "SINAI/ALIA-MrBERT-es-snomed-dental-ner-multiclass-ctx8192"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

text = "Paciente refiere abrasión dental y requiere un puente de dentina."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=8192)

with torch.no_grad():
    outputs = model(**inputs)

predictions = torch.argmax(outputs.logits, dim=-1)
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

for token, pred_id in zip(tokens, predictions[0].tolist()):
    label = model.config.id2label[pred_id]
    if label != "O":
        print(f"Token: {token:15} -> Label: {label}")

---

Evaluation

The model was evaluated using a train/test split containing 80% training and 20% testing data, representing a total of 6,022 entities in the test set.

Metrics

The token-level multiclass Named Entity Recognition performance on the test set is as follows:

---

Supported SNOMED CT Codes

This model is trained to recognize and directly code the following clinical terms from the Spanish dentistry domain:

---

Limitations and Biases

Known Limitations

• Large Label Space: Classifying tokens into ~815 possible classes (BIO tags for ~400 concepts) is a highly complex task. While it allows direct mapping, rare concepts with limited training representation may show lower extraction performance compared to more frequent terms like caries.
• Domain Focus: Highly specialized for Spanish dental reports and maxillofacial descriptions.
• Syntactic Context: Mappings rely on contextual clues. Out-of-context dental lists or non-standard abbreviations may be misclassified.

Biases

• Training data primarily mirrors institutional writing conventions and regional terminology variations from the clinic source.

---

Additional Information

License

Apache License, Version 2.0

Citation

If you use this model in your research, please cite:

@misc{ALIA-MrBERT-es-snomed-dental-ner-multiclass-ctx8192,
  title        = {ALIA MrBERT-es Snomed Dental Multiclass NER Model},
  author       = {SINAI Research Group, Universidad de Jaén},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/SINAI/ALIA-MrBERT-es-snomed-dental-ner-multiclass-ctx8192}}
}

Please also cite the base model:

@misc{tamayo2026mrbertmodernmultilingualencoders,
      title={MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation}, 
      author={Daniel Tamayo and Iñaki Lacunza and Paula Rivera-Hidalgo and Severino Da Dalt and Javier Aula-Blasco and Aitor Gonzalez-Agirre and Marta Villegas},
      year={2026},
      eprint={2602.21379},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.21379}, 
}

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ALIA.

Acknowledgments

This dataset has been generated thanks to CEATIC (Centro de Estudios Avanzados en Tecnologías de la Información y de la Comunicación) – UJA (Universidad de Jaén) which provided the needed computational resources on its clusters.

---

Contact: ALIA Project - SINAI Research Group - Universidad de Jaén

More Information: SINAI Research Group | ALIA-UJA Project