Downloads · 30 days
6
2% of all-time downloads
CardioBERTa_base.nl
UMCU/CardioBERTa_base.nl
CardioBERTa_base.nl is a fill-mask model from UMCU. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as gpl-3.0.
Continued, off-premise, pre-training of MedRoBERTa.nl using about 50GB of open Dutch and translated English corpora.
All-time downloads
297
Public
Parameters
166M
664 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors664 MB · 100%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- gpl-3.0
- Model type
- roberta
- Access
- Public
- Created
- Mar 4, 2025
- Updated
- Jun 21, 2025
- SHA
- 31286c20
Base models
Model card
From the Hugging Face model README
Continued, off-premise, pre-training of MedRoBERTa.nl using about 50GB of open Dutch and translated English corpora.
Data statistics
Sources:
- Dutch: medical guidelines (FMS, NHG)
- Dutch: NtvG papers
- English: Pubmed abstracts
- English: PMC abstracts translated using DeepL
- English: Apollo guidelines, papers and books
- English: Meditron guidelines
- English: MIMIC3
- English: MIMIC CXR
- English: MIMIC4
All translated (if not with DeepL) with a combination of GeminiFlash 1.5/GPT4o mini, MariaNMT, NLLB200.
- Number of tokens: 15B
- Number of documents: 27M
Training
- Effective batch size: 5120
- Learning rate: 2e-4
- Weight decay: 1e-3
- Learning schedule: linear, with 5_000 warmup steps
- Num epochs: ~3
Train perplexity: 3.0 Validation perplexity: 3.0
Acknowledgement
This work was done together with the Amsterdam UMC, in the context of the DataTools4Heart project.
We were happy to be able to use the Google TPU research cloud for training the model.
- Task
- Fill-Mask
- Library
- transformers
- Type
- roberta
- License
- gpl-3.0
- Languages
- nl
- Created
- Mar 4, 2025
- Updated
- Jun 21, 2025