Downloads · 30 days
12
12% of all-time downloads
jensjepsen/danish-lm-400m-base-ropext2048-v1
danish-lm-400m-base-ropext2048-v1 is a machine learning model from jensjepsen. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
RoPE-extended (512 → 2048 context) continuation of jensjepsen/danish-lm-400m-base-ckpt310k.
Downloads · 30 days
12
12% of all-time downloads
All-time downloads
99
Public
Parameters
415M
1.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.7 GB · 100%
From the Hugging Face model README
RoPE-extended (512 → 2048 context) continuation of jensjepsen/danish-lm-400m-base-ckpt310k.
Same architecture, same weights initialized from the base, then continued pretraining for ~160M tokens with rope_theta bumped from 10,000 → 500,000 and max_position_embeddings bumped from 512 → 2048. This is checkpoint-2500 of a planned 7,630-step run, stopped early at the empirical sweet spot before knowledge degradation began.
| base ckpt310k | this model | |
|---|---|---|
max_position_embeddings | 512 | 2048 |
rope_theta | 10,000 | 500,000 |
| architecture | Llama, 414M params, GQA 24/4, head_dim 64, hidden 1536, 16 layers, 16k byte-level BPE vocab | identical |
Only the RoPE rotation-frequency table changed; every attention/MLP weight was initialized from the base and then updated by ~160M tokens of continued pretraining.
jensjepsen/danish-lm-400m-base-ckpt310kjensjepsen/danish-pretokenized-16k, filtered to docs ≥2048 tokens (mostly Danish Wikipedia articles and books from Gutenberg/Internet Archive)rope_theta=500000 matches the "safe" value used by Llama-3.1 for its context extension.danish-pretrain not used for training)| positions | base ppl | this model | change |
|---|---|---|---|
| [0, 512) | 13.72 | 8.72 | -36% |
| [512, 2048) | 748 (base + dynamic NTK) | 17.42 | -98% |
Positions past 512 are meaningfully processed for the first time. The short range also improved, so the extension was not a knowledge trade-off.
Log-prob probe on alexandrainst/danish-citizen-tests (720 rows, MC cloze scoring):
| model | accuracy |
|---|---|
| base ckpt310k | 57.78% |
| this model | 57.64% |
−0.14pp = within noise for 720 rows. Extension came essentially free on Danish civic-knowledge recall.
Drop-in replacement for jensjepsen/danish-lm-400m-base-ckpt310k any time you need >512-token context: long-form SFT, tool-calling with catalogs, agentic multi-turn conversations, long wiki-grounded QA. Same tokenizer, same tokenizer chat template.
Trained via scripts/launch_da_ropext_2048.sh at commit a46ad6c, with --rope-extend-theta 500000 and --min-doc-tokens 2048. Full config at configs/da_400m_ropext_2048.yaml.
Selection of ckpt-2500 as the release checkpoint was empirical: run went to ckpt-4000, per-checkpoint held-out ppl + citizen-tests accuracy showed ckpt-2500 as the Pareto frontier (LM improved further at ckpt-3000-3500, but citizen-tests accuracy started dropping past 2500; ckpt-4000 regressed on both).