Downloads · 30 days
14
4% of all-time downloads
CardioMedRoBERTa.nl
UMCU/CardioMedRoBERTa.nl
CardioMedRoBERTa.nl is a fill-mask model from UMCU. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as gpl-3.0.
Continued, on-premise, pre-training of MedRoBERTa.nl using de-identified Electronic Health Records from the University Medical Center Utrecht, related to the cardiology domain.
All-time downloads
329
Public
Parameters
126M
1.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors504 MB · 100%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- gpl-3.0
- Model type
- roberta
- Access
- Public
- Created
- Feb 27, 2025
- Updated
- Jun 25, 2025
- SHA
- a320f009
Base models
Model card
From the Hugging Face model README
Continued, on-premise, pre-training of MedRoBERTa.nl using de-identified Electronic Health Records from the University Medical Center Utrecht, related to the cardiology domain.
Data statistics
Sources:
-
Dutch medical guidelines (FMS, NHG)
-
NtvG papers
-
PMC abstracts translated using GeminiFlash 1.5
-
Number of tokens: 1.47B, of which 1B from UMCU EHRs
-
Number of documents: 5.8M, of which 3.5M UMCU EHRs
-
Average number of tokens per document: 253
-
Median number of tokens per document: 124
Training
- Effective batch size: 240
- Learning rate: 1e-4
- Weight decay: 1e-3
- Learning schedule: linear, with 25_000 warmup steps
- Num epochs: 3
Train perplexity: 3.0
Validation perplexity: 4.0
Acknowledgements
This work was done together with the Amsterdam UMC, in the context of the DataTools4Heart project.
- Task
- Fill-Mask
- Library
- transformers
- Type
- roberta
- License
- gpl-3.0
- Languages
- nl
- Created
- Feb 27, 2025
- Updated
- Jun 25, 2025