ModernBERT-large-zeroshot-v2.0-ONNX

ModernBERT-large-zeroshot-v2.0 (ONNX)

This is an ONNX version of MoritzLaurer/ModernBERT-large-zeroshot-v2.0. It was automatically converted and uploaded using this Hugging Face Space.

Usage with Transformers.js

See the pipeline documentation for text-classification: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TextClassificationPipeline

---

ModernBERT-base-zeroshot-v2.0

Model description

This model is answerdotai/ModernBERT-large fine-tuned on the same dataset mix as the zeroshot-v2.0 models in the Zeroshot Classifiers Collection.

General takeaways:

• The model is very fast and memory efficient. It's multiple times faster and consumes multiple times less memory than DeBERTav3. The memory efficiency enables larger batch sizes. I got a ~2x speed increase by enabling bf16 (instead of fp16).
• It performs slightly worse then DeBERTav3 on average on the tasks tested below.
• I'm in the process of preparing a newer version trained on better synthetic data to make full use of the 8k context window and to update the training mix of the older zeroshot-v2.0 models.

Training results

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 9e-06
• train_batch_size: 16
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 2

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0