HanseLM-78M-Base

<h1 align="center"> HanseLM 78M Base</h1>

HanseLM 78M Base is a small German-first causal language model and an experimental research preview. It was trained from scratch to study whether a hybrid attention/convolution architecture can learn useful German text representations under a roughly 1.5-billion-token compute budget.

This is not a frontier model, not an instruction-tuned assistant, and not expected to be broadly useful without further post-training. It may produce incorrect, repetitive, mixed-language, unsafe, or memorized text. The release is primarily useful for architecture experiments, education, controlled fine-tuning, and reproducibility work.

The release code and model weights are available under the MIT License. The source datasets are not redistributed and retain their own terms; see DATA_SOURCES.md.

Model details

Architecture

Each block applies a pre-normalized token mixer and a pre-normalized SwiGLU feed-forward network with residual connections. The layer pattern is:

A A C A A A C A A A C A A A

A blocks use causal scaled-dot-product grouped-query attention. Ten query heads share two key/value heads in five-head groups. Queries and keys receive per-head RMS normalization before rotary position encoding. C blocks use a gated, depthwise causal 1D convolution with kernel size 7. Convolution blocks occur at layers 3, 7, and 11 (one-indexed). All projections are bias-free.

The hybrid design reduces the number of global-attention layers, but this release does not claim an inference-speed advantage: the reference implementation has no KV cache and currently recomputes the visible context during autoregressive generation.

Tokenizer

The tokenizer is a custom NFC-normalizing, byte-level BPE tokenizer with byte fallback. It was trained alongside the project and contains:

<|pad|> <|bos|> <|eos|> <|system|> <|user|>
<|assistant|> <|tool|> <|tool_result|>

The presence of role tokens does not make the base model chat-capable. No chat template is defined for this repository.

Training

The sampled pretraining mixture contained approximately 1.500 billion tokens:

Training used two stages:

1. 1.350B tokens with 1,024-token sequences.
2. 150.012M additional tokens with 2,048-token sequences.

Phase 2 continued from the Phase 1 final checkpoint with AdamW (betas=(0.9, 0.95)), a peak learning rate of 4e-5, a minimum learning rate of 4e-6, and 100 warmup steps. Training ran locally with PyTorch/ROCm on an AMD Radeon RX 9070 XT.

Compute and hardware

The full base model was trained locally on a single consumer GPU, not a multi-GPU server or cloud cluster:

Of the total, 8.01 hours come directly from structured trainer timers and approximately 1.4 hours are estimated for the initial unstructured segment. The Phase 1 number includes replayed work after interrupted runs. Timers include validation and checkpoint overhead while the trainer was active, but exclude intentional pauses, dataset preparation, checkpoint selection, and the later release evaluations.

This modest hardware footprint is part of the experiment: HanseLM tests what can be trained end-to-end on one 16 GB desktop GPU. It should not be confused with frontier-scale pretraining, and the low compute budget is also a major reason for the model's limited factual and reasoning ability.

Documents were packed with EOS separators. The current attention mask does not isolate documents inside a packed sequence, so tokens can attend across an EOS boundary. This is a known training limitation.

Evaluation

The selected final checkpoint beat the late Phase 2 checkpoints on three independent mixed-validation samples. The margin over step 12,000 was small but consistent.

Held-out language-model loss

All rows below use 2,048-token sequences, batch size 6, 200 batches, and seed 1,234.

Perplexity is specific to this tokenizer and each corpus. Values must not be compared directly with models using another tokenizer, and these results are not downstream knowledge, reasoning, safety, or instruction-following benchmarks. Standardized German and English benchmarks have not yet been run.

Phase 2 improved the shared mixed-validation loss from 3.228526 (Phase 1) to 3.175663.

A fixed-seed generation check found no repeated token trigrams in the two German and one English sample; a deliberately mixed prompt had a 0.0217 repeated-trigram fraction and continued in English. These four samples are too small for a quality claim. They also exposed a clear factual error about Hamburg, reinforcing that the model must not be used as a factual source. The full samples and a successful 2,048-token finite-logit smoke test are recorded under evaluation/.

A broader 20-sample suite (10 prompts, two seeds) produced a mean repeated token-trigram fraction of 0.0445 and a maximum of 0.1613. German and English continuations were often locally grammatical, and the mixed prompt continued in German. However, factual prompts failed consistently: the model called Berlin the capital of southern France or Croatia, placed German federal history in 1921/1998, and gave contradictory freezing temperatures. Recipe and story continuations were recognizable in form but frequently incoherent. The honest conclusion is that the checkpoint has learned language form and domain-like continuation, but not dependable knowledge or instruction following. Raw outputs are stored in evaluation/generation-suite.json.

Usage

This repository contains custom Transformers code, so loading requires explicitly trusting the repository code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/HanseLM-78M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Die Hanse war", return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_k=50,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The reference implementation does not implement KV caching. Generation is therefore slower than similarly sized cached Transformer models.

Intended uses

• Research on small language models and hybrid token mixers.
• Reproducible continued pretraining or post-training experiments.
• Educational inspection of a complete from-scratch model pipeline.
• Low-stakes, human-reviewed German or English text continuation experiments.

It is not intended as a factual source, production assistant, autonomous agent, safety-critical component, or replacement for a larger evaluated model. High-impact decisions and unsupervised public text generation are out of scope.

Limitations and risks

• Only 78M parameters and about 1.5B training tokens.
• Mostly German web text; only 5% of the sampled mixture was English.
• No instruction tuning, preference optimization, or safety post-training.
• No systematic benchmark, bias, toxicity, privacy, or memorization audit.
• No guaranteed factuality, reasoning, arithmetic, code, or tool-use ability.
• Possible repetition, abrupt continuations, malformed text, and language switching.
• A maximum trained context of 2,048 tokens; longer inputs are rejected.
• Packed-document cross-attention as described above.
• Web and Wikipedia data can contain errors, stereotypes, personal data, copyrighted material, and other undesirable content.

Reproducibility

The staged evaluation record is in evaluation/internal-eval.json. The selected weight source is the Phase 2 final checkpoint after 150,011,904 Phase 2 tokens. The model contains 78,430,848 parameters.

The custom adapter was verified with Transformers 4.57.6 and PyTorch 2.12.0+rocm7.14.0. Loading through AutoModelForCausalLM produced no missing or unexpected keys, and its logits matched the native implementation exactly for the equivalence input (max_abs_diff = 0). Artifact checksums are recorded in MANIFEST.sha256.

License and attribution

The HanseLM release code and model weights are licensed under the MIT License. Training-source attribution and third-party terms are documented separately in DATA_SOURCES.md. In particular, FineWeb and FineWeb2 identify ODC-By 1.0, while FineWiki identifies CC BY-SA 4.0 and GFDL for its Wikipedia-derived text. The MIT license does not relicense those source datasets.

Citation

No paper or archival report exists yet. For reproducible references, cite the repository and an immutable release revision:

@software{hanse_lm_78m_base_2026,
  author = {EvickaStudio},
  title = {HanseLM 78M Base},
  year = {2026},
  url = {https://huggingface.co/Evicka/HanseLM-78M-Base},
  version = {v0.1}
}