Downloads · 30 days
87
11% of all-time downloads
birgermoell/oellm-9b-128k-theta32m-prelude-anneal300b
oellm-9b-128k-theta32m-prelude-anneal300b is a text generation model from birgermoell. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is a BF16 Hugging Face export of a 128K continued-pretraining checkpoint derived from openeurollm/prelude, revision f289699b246dba59907df27e743d4a433613175a (anneal300biter0989075). It uses the Qwen3 dense 9B arc…
Downloads · 30 days
87
11% of all-time downloads
All-time downloads
824
Public
Repo size
18.2 GB
Likes
0
Public
Click a slice to open those files.
.bin18.2 GB · 100%
From the Hugging Face model README
This is a BF16 Hugging Face export of a 128K continued-pretraining checkpoint derived from
openeurollm/prelude, revision
f289699b246dba59907df27e743d4a433613175a (anneal300b_iter_0989075). It uses the Qwen3 dense
9B architecture, a 131,072-token maximum position length, and RoPE theta 32,000,000.
This is a base completion model. It is not instruction-tuned or safety-aligned.
The goal of this run was to test whether the later anneal300b Prelude checkpoint could be
extended from its native short context to 128K while preserving useful retrieval at shorter
lengths. It follows the same staged absolute-base-frequency (ABF) recipe as the earlier
birgermoell/oellm-9b-128k-theta32m-prelude,
but it is a different lineage:
openeurollm/prelude@anneal300b_iter_0989075, the approximately 300B-token
annealed checkpoint at commit f289699b246dba59907df27e743d4a433613175a.The separate repository name is intentional. This model is an experimental comparison point, not an in-place replacement for the earlier 1T-base release.
| Property | Value |
|---|---|
| Architecture | Qwen3-compatible dense decoder-only transformer |
| Parameters | approximately 9B |
| Transformer layers | 36 |
| Hidden size | 4,096 |
| Feed-forward size | 12,288 |
| Attention heads | 32 |
| Key/value groups | 8 (grouped-query attention) |
| Head dimension | 128 |
| Vocabulary | 262,144 tokens |
| Normalization | RMSNorm with Q/K layer normalization |
| MLP | SwiGLU |
| Embeddings | untied input/output embeddings |
| Maximum positions | 131,072 |
| RoPE theta | 32,000,000 |
| Weight dtype | BF16 |
The tokenizer is the OpenEuroLLM 256K-vocabulary tokenizer. The exported Transformers configuration preserves the Q/K normalization parameters and the 128K positional settings used during training.
The model was continued on LUMI using staged native ABF context extension:
| Stage | Sequence length | RoPE theta | Iterations | Tokens | LUMI job |
|---|---|---|---|---|---|
| 16K | 16,384 | 500,000 | 953 | 999,292,928 | 21683851 |
| 32K | 32,768 | 1,000,000 | 476 | 998,244,352 | 21683852 |
| 64K | 65,536 | 2,000,000 | 715 | 2,998,927,360 | 21740962 |
| 128K | 131,072 | 32,000,000 | 238 | 1,996,488,704 | 21683854 |
Total continued-pretraining volume through this checkpoint: 6,992,953,344 tokens.
The curriculum increases both sequence length and RoPE theta at each stage:
Prelude anneal300b
-> 16K (theta 500K)
-> 32K (theta 1M)
-> 64K (theta 2M)
-> 128K (theta 32M)
The unusually large final theta is deliberate. At long retrieval distances, smaller theta values can leave high-frequency RoPE dimensions outside the range encountered during training. The theta schedule follows the behavior characterized in the earlier Prelude long-context work; the new quick evaluation below tests whether it transfers to this later base checkpoint.
1e-5 to 1e-6, with approximately 5% warmup.torch_dist; optimizer and RNG state were not
retained in the release export.The 128K stage completed all 238 iterations on its first launcher attempt, saved the final checkpoint successfully, reported validation loss 1.556005 (perplexity 4.73985), and had zero skipped and zero NaN iterations. These values are run-health diagnostics, not cross-model benchmark scores.
The 16K–128K curriculum used a frozen, token-proportional multilingual blend with 152 data prefixes. It is length-aware and draws from long and general-domain sources including FinePDFs, DCLM, HPLT3, multilingual synthetic material, Nemotron data, MegaMath, StarCoder, PeS2o, arXiv, and Wikipedia. Sources were grouped into short, medium, and long tiers so that longer natural documents were represented as sequence length increased.
The mixture spans European languages and code/scientific text, but it is not balanced equally by language. The model therefore should not be interpreted as having uniform capability across every language listed in the metadata. The frozen 16K–128K blend hash is provided below for exact run identity. The exact 128K training receipt and the machine-readable quick-evaluation records are included in this repository.
The iteration-238 Megatron checkpoint was converted directly to a Qwen3-compatible Transformers layout in BF16. Publication is gated on all of the following:
model_type=qwen3, max_position_embeddings=131072, and
rope_theta=32000000.The release job ran a deliberately small, base-LM forced-choice needle-in-a-haystack smoke test at 16K, 32K, 64K, and 128K. Each language/context cell contains three needle depths and two trials per depth (six cases). This verifies export and retrieval behavior; it is too small for a publication-grade accuracy estimate.
| Context | English | Swedish | Combined |
|---|---|---|---|
| 16,384 | 100.0% (6/6) | 100.0% (6/6) | 100.0% (12/12) |
| 32,768 | 100.0% (6/6) | 100.0% (6/6) | 100.0% (12/12) |
| 65,536 | 100.0% (6/6) | 100.0% (6/6) | 100.0% (12/12) |
| 131,072 | 100.0% (6/6) | 100.0% (6/6) | 100.0% (12/12) |
Depth 0.0 places the needle at the far beginning (maximum retrieval distance); depth 1.0 places it nearest the query.
| Language | Context | Depth 0.0 | Depth 0.5 | Depth 1.0 |
|---|---|---|---|---|
| en | 16,384 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| en | 32,768 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| en | 65,536 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| en | 131,072 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| sv | 16,384 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| sv | 32,768 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| sv | 65,536 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| sv | 131,072 | 100.0% (2/2) | 100.0% (2/2) | 100.0% (2/2) |
| Language | Control | Accuracy |
|---|---|---|
| en | short_ctx | 100.0% (2/2) |
| en | shuffled | 100.0% (2/2) |
| en | no_context | 50.0% (1/2) |
| sv | short_ctx | 100.0% (2/2) |
| sv | shuffled | 100.0% (2/2) |
| sv | no_context | 0.0% (0/2) |
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "birgermoell/oellm-9b-128k-theta32m-prelude-anneal300b"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
Long-context inference requires substantial accelerator memory. Keep rope_theta=32000000 and
max_position_embeddings=131072 unchanged for the trained 128K configuration.
This checkpoint is best used for base-model research, continued pretraining, long-context retrieval experiments, and as a starting point for subsequent supervised or preference-based post-training. It is not a chat model; use raw completion prompts or a base-LM evaluation harness.
For shorter prompts, no special scaling override is needed: the model should be loaded with its published 128K configuration and evaluated directly at the desired shorter length.
The included smoke test uses forced-choice needle-in-a-haystack scoring designed for base models. Each context contains key/value facts. A target fact is placed at a selected relative depth, and the model scores four candidate values by answer-token log likelihood. Distractors are values of other keys that really occur in the same context, making the test stricter than choosing against arbitrary out-of-context numbers. The highest-scoring candidate is the prediction.
The test covers English and Swedish at nominal 16K, 32K, 64K, and 128K lengths. Actual context lengths can be slightly below the nominal boundary because the generator fills contexts using complete fact lines and leaves room for the query and candidate suffix. Depth 0.0 is the most distant case: the needle is at the beginning and the query is at the end.
Controls check that short-context scoring works, that the model follows the presented binding
rather than a memorized key/value association, and that the no-context condition remains near
chance. All per-case records are published under evaluation/.
f289699b246dba59907df27e743d4a433613175ab359462c12858cedd2238a22eca0dca7aa6b8872ac214401227404105817141c21c6693e3b17c878465a42765d2c345310e003cec25d345c2c1969be4308fab2c5423584ef2fade72b1e711558fd459936b7ac6abirgermoell/oellm-9b-128k-theta32m-preludeopeneurollm/preludeBirgerMoell/openeuro-longctx-datamix