german-commons-350m

German Commons 350M, step 10,000

Author: Hamid Wakili
Contact: [email protected]
Source code: ideployed/sovereign-german-llm

This repository contains an intermediate checkpoint of my German base-model experiment. I trained the tokenizer and model from random initialization. The release does not use an imported tokenizer vocabulary, pretrained weights, distillation, or synthetic training text.

The checkpoint was saved after 10,000 optimizer steps, equivalent to 5,242,880,000 training tokens at the configured global batch size. The prepared corpus contains 30.00B training tokens, but this checkpoint has not completed that training target.

This is a base model for text completion, not a chatbot. It has not been instruction-tuned and should not be expected to answer questions or follow commands reliably.

Release contents

• model.safetensors: inference weights from step 10,000
• config.json: architecture configuration
• german_transformer_model.py: custom PyTorch model implementation
• bpe_tokenizer.py: custom byte-level BPE implementation
• inference.py: offline Safetensors generation CLI
• tokenizer/de_bpe_32k/: matching vocabulary, merges, and special tokens
• samples/generations_5k_vs_10k.md: deterministic checkpoint comparison
• samples/training_curves/: loss, perplexity, optimization, and throughput plots
• SHA256SUMS: integrity hashes for the model, runtime code, and tokenizer

The optimizer and scheduler states are intentionally excluded. The original resumable checkpoint is a separate training artifact.

Model architecture

I implemented the architecture directly with PyTorch. It is not registered with the Hugging Face Transformers AutoModel classes, so trust_remote_code and AutoModelForCausalLM are not supported by this release. The complete training and data-preparation code is available in the GitHub repository.

Tokenizer

The tokenizer is a byte-level BPE tokenizer trained for this experiment on a sample from German Commons. Its fixed IDs are:

IDs 5 through 260 represent the 256 byte values. Higher IDs represent learned BPE merges. The tokenizer uses </w> as an explicit word boundary marker.

Training data

The prepared data was streamed from German Commons, quality-filtered, domain-reweighted, and encoded into local uint16 shards. The prepared dataset contains:

• 30,000,000,243 training tokens
• 50,002,350 validation tokens
• a stable-hash document split with seed 1337
• aggregate source and license counts recorded during preparation

The accepted-document license distribution recorded in the preparation manifest was approximately 51.6% CC0-1.0, 46.2% CC-BY-4.0, and 2.2% CC-BY-SA-4.0. These percentages count accepted documents, not tokens.

German Commons contains a substantial amount of historical newspaper and cultural material. Domain reweighting reduced that concentration but did not remove it. The resulting model frequently uses historical spelling and register.

Training configuration

At step 10,000, the recorded training loss was 3.2307. Evaluation over 20 batches from the held-out validation shards produced a validation loss of 3.2593 and perplexity of 26.03. This is a training diagnostic, not a broad language-model benchmark.

Evaluation status

No standardized downstream benchmark results are reported for this checkpoint. The available evaluation consists of held-out German Commons loss and direct inspection of deterministic completions. The 5k and 10k comparison is published in samples/generations_5k_vs_10k.md.

Those samples are intentionally unedited. They show improved German structure at step 10,000, but also repetition, historical register, and severe factual fabrication.

Usage

Install PyTorch and Safetensors, clone or download this repository, and run from its root:

python inference.py \
  --model-dir . \
  --prompt "Die deutsche Sprache ist" \
  --max-new-tokens 80 \
  --temperature 0.8 \
  --top-k 40 \
  --top-p 0.9

The underlying loading path is plain PyTorch:

import json

import torch
from safetensors.torch import load_file

from bpe_tokenizer import BPE_Tokenizer
from german_transformer_model import GermanGPT, GermanGPTConfig


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

with open("config.json", "r", encoding="utf-8") as file:
    config = GermanGPTConfig.from_dict(json.load(file))

tokenizer = BPE_Tokenizer.load("tokenizer/de_bpe_32k")
model = GermanGPT(config)
state_dict = load_file("model.safetensors", device="cpu")
model.load_state_dict(state_dict, strict=True)
model.to(device).eval()

prompt = "Die deutsche Sprache ist"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long, device=device)

with torch.inference_mode():
    logits, _ = model(input_ids)

next_token = int(torch.argmax(logits[0, -1]).item())
print(tokenizer.decode(input_ids[0].tolist() + [next_token]))

This example performs one greedy next-token step. Autoregressive sampling can be implemented by repeatedly passing the most recent 2,048 tokens through the model and applying temperature, top-k, or top-p filtering.

Intended use

• Research on German base-model pretraining and tokenization
• Inspection of training behavior at an early checkpoint
• Continuation training and controlled fine-tuning experiments
• Non-factual German text-completion experiments

Limitations

1. Intermediate checkpoint. The model has consumed about 17.5% of the prepared 30B-token target.
2. Not instruction-tuned. Question answering and chat prompts are outside the model's training objective.
3. Unreliable facts. The model confidently invents people, places, dates, quantities, and relationships. Its output must not be treated as evidence.
4. Repetition. Longer generations often enter repeated phrases or sentence structures.
5. Historical register. Archaic spelling and phrasing occur frequently due to the composition of the accepted training documents.
6. German-focused. Other languages were not evaluated and are outside the intended scope.
7. Inherited corpus bias. Filtering and provenance records do not remove social, historical, geographic, or source-selection bias.
8. No safety evaluation. I have not completed red-team, memorization, privacy, toxicity, or demographic-bias evaluations.

Do not use this checkpoint for medical, legal, financial, emergency, or other high-impact decisions. It is not suitable for autonomous decision-making about people.

Licensing and attribution

• The project code in this repository is licensed under the MIT License; see LICENSE.
• The model weights are released under CC BY-SA 4.0 as stated in the repository metadata.
• German Commons and each constituent source retain their own attribution and licensing requirements. Users are responsible for reviewing those terms for their intended use.

Please attribute both Hamid Wakili for this model release and German Commons as the training-data source.

Citation

@misc{wakili2026germancommons350m,
  author = {Hamid Wakili},
  title = {German Commons 350M, Step 10,000},
  year = {2026},
  note = {Intermediate German base-model checkpoint},
  url = {https://github.com/ideployed/sovereign-german-llm}
}