ModernCamemBERT-bio-v2-base

<a href=https://camembert-bio-model.fr/> <img width="210px" src="./camembert-bio-logo.png"> </a>

ModernCamemBERT-bio-v2-base

ModernCamemBERT-bio-v2-base is a French biomedical encoder for document-level tasks. It is obtained by continuing the pretraining of ModernCamemBERT-bio-base with an OntoBook phase, with an 8,192-token context window. On our French biomedical document benchmark it reaches 70.0 micro-F1 at 150M parameters. A larger version is available as ModernCamemBERT-bio-v2-large.

Authors

Rian Touchent and Eric de la Clergerie
Sorbonne Université, Inria Paris

Overview

ModernCamemBERT-bio is already adapted to French biomedical text through a CLM detour. The v2 variant adds a second continued-pretraining phase that we call OntoBook. Instead of plain masked language modeling, we train on ontology-grounded synthetic textbooks built from CIM-10, CCAM and ATC ontology walks, mixed with clinical text. The aim is to inject structured coding knowledge without adding a task-specific head. This is the base-size document specialist.

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "rntc/ModernCamemBERT-bio-v2-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

text = "Le patient présente une [MASK] aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))

Evaluation

Per-task results on the document core of our French biomedical benchmark, scored with micro-F1 and averaged over nine seeds.

At base size, ModernCamemBERT-bio-v2-base leads the average and scores 1.4 points above its ModernCamemBERT-bio-base starting point. ModernCamemBERT-bio-v2-large scores 2.8 points higher at 350M parameters.

Environmental impact

The OntoBook phase reported here ran for about 1.3 GPU-hours on a single H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.04 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.

License

MIT

Citation

@inproceedings{touchent:hal-05697506,
  TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
  AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
  URL = {https://hal.science/hal-05697506},
  BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
  ADDRESS = {Palma de Mallorca, Spain},
  YEAR = {2026},
  MONTH = May,
  PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
  HAL_ID = {hal-05697506},
  HAL_VERSION = {v1},
}

@misc{touchent2026causallanguagemodelingdetour,
  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
  author={Rian Touchent and Eric de la Clergerie},
  year={2026},
  eprint={2605.12438},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2605.12438},
}