CT-ModernBERT-JPN

CT-ModernBERT-JPN

CT-ModernBERT-JPN is a Japanese ModernBERT-based model for multilabel classification of abnormal findings in radiology reports, fine-tuned on the CT-RATE-JPN dataset. This model offers superior tokenization efficiency and training speed compared to conventional BERT models while maintaining comparable classification performance.

Model Overview

This model is based on ModernBERT Japanese 130M, and has been fine-tuned on the CT-RATE-JPN dataset, which provides Japanese translations of radiology reports from the CT-RATE dataset. The training data consists of deduplicated radiology reports with corresponding abnormality labels for 18 common chest CT conditions.

Key Advantages of ModernBERT

• Superior Tokenization Efficiency: Requires 24.0% fewer tokens per document compared to BERT Base
• Faster Training: Completes training 39% faster than BERT Base
• Improved Processing Speed: 1.65× faster during training and 1.66× faster during inference
• Comparable Classification Performance: Achieves similar or slightly better performance across most conditions

How to Use

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load the model and tokenizer from Hugging Face Hub
model_name = "YYama0/CT-ModernBERT-JPN"
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=18,
    problem_type="multi_label_classification"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Define the inference function
def infer(input_texts):
    inputs = tokenizer(input_texts, padding=True, truncation=True, return_tensors="pt")
    model.eval()
    with torch.no_grad():
        outputs = model(**inputs)
    logits = outputs.logits
    probs = torch.sigmoid(logits)
    return probs

# Run inference
input_texts = ["気管および両主気管支の内腔は開存しています。気管および両主気管支の内腔には閉塞病変は認められませんでした。縦隔内の主要血管構造、心臓の輪郭、サイズは正常です。胸部大動脈の直径は正常です。心嚢水、心膜肥厚は確認されませんでした。胸部食道径は正常であり、非造影検査の範囲内で有意な病的壁肥厚は認められませんでした。縦隔内の上下部気管傍、大動脈肺動脈窓の血管前領域、および気管分岐下において、最大短軸が7mmのリンパ節が認められました。肺野条件では、右側の胸膜葉の間に最大で8cmの厚さに達する広範な胸水が認められました。左側では、最も広い部分で26mmです。隣接する肺実質には、特に右側でびまん性の無気肺変化が認められました。加えて、両肺で小葉間隔壁の肥厚を伴うびまん性のすりガラス陰影の増加およびcrazy paving appearancesが認められました。これらの所見は感染症過程と一致している可能性があります。肺水腫も鑑別診断に考慮されるべきです。臨床および検査との対比を考慮すること、および治療後の管理が推奨されます。両肺にミリ単位の非特異的な実質性結節がいくつか認められました。検査範囲の上腹部では、有意な病変は認められませんでした。骨構造においても溶骨性破壊病変は認められませんでした。"]
probs = infer(input_texts)

Model Performance

The following table shows the F1 scores for each medical condition on the validation data (n=150) of CT-RATE-JPN, compared with the BERT Base model. For metrics requiring binarization, a threshold of 0.5 was used.

Overall Exact Match Accuracy: 74.67% (vs. 72.67% for BERT Base)

Efficiency Comparison

Citation

Base ModernBERT Model:

Please cite the original ModernBERT Japanese model from sbintuitions/modernbert-ja-130m.

ModernBERT Architecture:

@misc{warner2024smarterbetterfasterlonger,
      title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference}, 
      author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
      year={2024},
      eprint={2412.13663},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.13663}, 
}

CT-RATE Dataset:

Please visit the original CT-RATE repository for the appropriate citation information.

CT-RATE-JPN and CT-ModernBERT-JPN:

@article{yamagishi2025ctrep,
  title   = {Development of a Large-Scale Dataset of Chest Computed Tomography Reports in Japanese and a High-Performance Finding Classification Model: Dataset Development and Validation Study},
  author  = {Yamagishi, Yosuke and Nakamura, Yuta and Kikuchi, Tomohiro and Sonoda, Yuki and Hirakawa, Hiroshi and Kano, Shintaro and Nakamura, Satoshi and Hanaoka, Shouhei and Yoshikawa, Takeharu and Abe, Osamu},
  journal = {JMIR Medical Informatics},
  year    = {2025},
  volume  = {13},
  pages   = {e71137},
  doi     = {10.2196/71137},
  url     = {https://doi.org/10.2196/71137}
}

For the comparative study between BERT Base and ModernBERT on Japanese radiology reports (which produced this model), please cite:

@article{yamagishi2026modernber,
  title   = {ModernBERT is more efficient than conventional BERT for chest CT findings classification in Japanese radiology reports},
  author  = {Yamagishi, Yosuke and Kikuchi, Tomohiro and Hanaoka, Shouhei and Yoshikawa, Takeharu and Abe, Osamu},
  journal = {Scientific Reports},
  year    = {2026},
  volume  = {16},
  pages   = {15956},
  doi     = {10.1038/s41598-026-44292-z},
  url     = {https://doi.org/10.1038/s41598-026-44292-z}
}

License

This model was trained using the CT-RATE-JPN dataset, which is released under the Creative Commons Attribution-NonCommercial-ShareAlike (CC BY-NC-SA) license.

Regarding model usage:

• The model outputs and weights can be used for non-commercial research purposes only
• When using the dataset, users must comply with the terms of the original CC BY-NC-SA license

Acknowledgments

• The original ModernBERT Japanese model developers (SB Intuitions)
• The CT-RATE dataset creators
• The CT-RATE-JPN dataset creators