Downloads · 30 days
24
28% of all-time downloads
de-t5-sci-transfer-15k
rausch/de-t5-sci-transfer-15k
de-t5-sci-transfer-15k is a text generation model from rausch. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
Final German scientific model: WECHSEL-initialized EN-T5-Sci → continued for 15 000 steps on German scientific data (same regimen as DE-T5-Base-15k). Checkpoint: crosslingualtransfer/logs/train/.../step-step=015000.ckpt.
All-time downloads
86
Public
Parameters
619M
2.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.5 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- t5
- Access
- Public
- Created
- Nov 17, 2025
- Updated
- Nov 17, 2025
- SHA
- 2fddeac2
Datasets
Model card
From the Hugging Face model README
DE-T5-Sci-Transfer-15k
Final German scientific model: WECHSEL-initialized EN-T5-Sci → continued for 15 000 steps on German scientific data (same regimen as DE-T5-Base-15k). Checkpoint: cross_lingual_transfer/logs/train/.../step-step=015000.ckpt.
Model Details
- Base: EN-T5-Sci weights, German tokenizer
- Optimizer: Adafactor, lr=1e-3, inverse sqrt schedule, warmup 1.5k, grad clip 1.0
- Effective batch: 48 (per-GPU 48, grad accumulation 1)
- Objective: Span corruption (15 % noise, mean span 3)
Training Data
German subset of the Unpaywall-derived scientific corpus (sliding windows 512 tokens, 50 % overlap). Same cleaning pipeline as the English run.
Evaluation (Global-MMLU, zero-shot)
| Metric | EN | DE |
|---|---|---|
| Overall accuracy | 0.2738 | 0.2700 |
| Humanities | 0.2559 | 0.2536 |
| STEM | 0.2867 | 0.2851 |
| Social Sciences | 0.3058 | 0.3055 |
| Other | 0.2562 | 0.2443 |
This is the best-performing German checkpoint across both languages in the final evaluation (evaluation_results/scientific_crosslingual_transfer_eval_full_15k).
Intended Use
Zero-shot QA in German/English scientific domains, or as a strong starting point for German task-specific fine-tuning (NER, relation extraction, etc.).
Limitations
- Still inherits T5-base context length/parameter budget.
- Evaluated only on Global-MMLU; downstream fine-tuning recommended for specialized tasks.
- Training corpus is domain-specific (scientific); may underperform on casual text.
Citation
Please cite the thesis (Nikolas Rauscher, 2025) and the WECHSEL paper (Minixhofer et al. 2022).
- Task
- Text Generation
- Library
- transformers
- Type
- t5
- License
- mit
- Languages
- de
- Created
- Nov 17, 2025
- Updated
- Nov 17, 2025