Downloads · 30 days
21
21% of all-time downloads
de-t5-base-continued-15k
rausch/de-t5-base-continued-15k
de-t5-base-continued-15k is a text generation model from rausch. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
GermanT5/t5-efficient-gc4-german-base-nl36 continued for 15 000 steps on the German portion of the scientific corpus (same preprocessing as EN). Checkpoint: crosslingualtransfer/logs/nativebaseline/.../step-step=01500…
All-time downloads
102
Public
Parameters
619M
2.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.5 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- t5
- Access
- Public
- Created
- Nov 17, 2025
- Updated
- Nov 17, 2025
- SHA
- 238d2881
Datasets
Model card
From the Hugging Face model README
DE-T5-Base-15k
GermanT5/t5-efficient-gc4-german-base-nl36 continued for 15 000 steps on the German portion of the scientific corpus (same preprocessing as EN). Checkpoint: cross_lingual_transfer/logs/native_baseline/.../step-step=015000.ckpt.
Model Details
- Base: GermanT5 (same architecture as T5-base, German tokenizer)
- Optimizer: Adafactor, lr=1e-3, inverse sqrt schedule, warmup=1.5k, grad clip=1.0
- Effective batch: 48 (per-GPU 48, no accumulation)
- Objective: Span corruption (15 % masking, mean span length 3)
Training Data
German split of the Unpaywall-derived corpus (continued-pretraining windows of 512 tokens, 50 % overlap).
Evaluation (Global-MMLU, zero-shot)
| Metric | EN | DE |
|---|---|---|
| Overall accuracy | 0.2295 | 0.2295 |
| Humanities | 0.2421 | 0.2421 |
| STEM | 0.2125 | 0.2125 |
| Social Sciences | 0.2171 | 0.2171 |
| Other | 0.2398 | 0.2398 |
Intended Use
German scientific NLP baseline; compare against WECHSEL-based models or continue fine-tuning on German datasets.
Limitations
- Only 15k steps, so improvements over base GermanT5 are modest.
- Uses German SentencePiece vocab; incompatible with English tokenizer out of the box.
- Task
- Text Generation
- Library
- transformers
- Type
- t5
- License
- mit
- Languages
- de
- Created
- Nov 17, 2025
- Updated
- Nov 17, 2025