Boldt-1B

Boldt-1B

<img src="logo.png" width="500">

Boldt is a series of German Small Language Models (SLMs) trained from scratch. Our inital release includes four models:

• Boldt-DC-350M
• Boldt-DC-1B
• Boldt-1B (this model)
• Boldt-1B-IT-Preview

Repetition over Diversity

The training philosophy behind Boldt is centered on a key finding from our research: repetition over diversity.

Standard pre-training paradigms typically balance quality filtering against the need for massive token volume and broad corpus diversity. In contrast, Boldt models are trained for multiple epochs on a highly filtered dataset: the German Dense-Core subset of FineWeb-2. We isolated this subset using a combination of three hierarchical filters:

• Coherence: Eliminates structurally fragmented or incoherent documents.
• Information Value: Isolates content-rich and fact-bearing texts.
• Educational Quality: Selects strictly for pedagogical clarity and deep explanations.

We demonstrate that repeated exposure to this strict, high-quality subset is more sample-efficient than a single pass over less filtered and more diverse corpora. For a comprehensive look at our experiments, please refer to our preprint: Repetition over Diversity.

Boldt-1B builds upon the foundation of Boldt-DC-1B by adding 6B tokens of premium German news data, collected continuously since 2022 via the Fundus library. To support complex downstream tasks, it also features a doubled context window of 4096 tokens.

Model Architecture

• Parameters: ~1 Billion
• Context Window: 4096 tokens
• Training Data: German Dense-Core subset (FineWeb-2) + 6B tokens high-quality Fundus news data
• Language: German

Usage

Note: This is a base language model, not an instruction-tuned model. It is not optimized for chat or instruction following. For best results, use standard text completion rather than chat templates.

from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "Boldt/Boldt-1B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Basic text completion
text = "Berlin ist eine Stadt, wo"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64)

Evaluation

We evaluate Boldt-1B on our modernized German benchmark suite. See our paper (Aynetdinov et al., 2026) for details on the structural and translation corrections we performed.

Despite being trained on substantially fewer tokens, the Boldt-1B family outperforms other 1B-class models on German tasks and performs competitively with much larger multilingual models.

1B Weight Class (Direct Comparison)

Note: Bold text indicates the best score in the 1B category.

1.7B - 2B Weight Class (Larger Reference Models)

Safety & Ethics

We have not conducted systematic model evaluations of toxicity, demographic biases, or harmful stereotypes. Quality filtering may reduce some risks relative to unfiltered web data, but cannot guarantee their absence, and repeated exposure during multi-epoch training could amplify rather than mitigate encoded biases. Users should exercise caution in sensitive use-cases without further evaluation.

Citation

@misc{boldt2026,
      title={Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling}, 
      author={Ansar Aynetdinov and Patrick Haller and Alan Akbik},
      year={2026},
      eprint={2604.28075},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.28075}, 
}