Downloads · 30 days
273
49% of all-time downloads
Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft
oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft is a text generation model from Neonkraft. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a full-parameter supervised fine-tune (SFT) of birgermoell/oellm-9b-256k-theta64m-prelude-anneal300b on allenai/Dolci-Instruct-SFT, the instruction mixture that Ai2 used for Olmo 3 7B Instruct SFT. It wa…
Downloads · 30 days
273
49% of all-time downloads
All-time downloads
558
Public
Parameters
9.1B
18.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors18.2 GB · 100%
From the Hugging Face model README
This model is a full-parameter supervised fine-tune (SFT) of
birgermoell/oellm-9b-256k-theta64m-prelude-anneal300b
on allenai/Dolci-Instruct-SFT, the
instruction mixture that Ai2 used for Olmo 3 7B Instruct SFT. It was trained for 2 epochs with
sequences of up to 32,768 tokens.
This is an experimental research checkpoint. It has had SFT only, with no preference tuning and no reinforcement learning.
Rows with tool use were rendered incorrectly during SFT of this model (missing tool information and empty tool use rows), so please expect poor capacity for tool use from this model.
The model uses the ChatML-style turn format of Qwen3. The tokenizer is the OpenEuroLLM 256K
tokenizer from its qwen3-tokens branch. That branch gives Qwen3 names to eight token IDs that the
base tokenizer had reserved:
| Token | ID | Name in the base tokenizer |
|---|---|---|
<|im_start|> | 3 | <start_of_turn> |
<|im_end|> | 4 | <end_of_turn> |
<tools> / </tools> | 13 / 14 | <unused_0> / <unused_1> |
<tool_response> / </tool_response> | 15 / 16 | <unused_2> / <unused_3> |
</think> / <think> | 17 / 18 | <unused_4> / <unused_5> |
Unchanged special tokens: <bos> = 1, <eos> = 2, <tool_call> / </tool_call> = 11 / 12,
<pad> = 262,144. All other token IDs are identical to the base tokenizer. On a sample of 3.9M
tokens of Dolci text, both tokenizers produced identical token IDs.
A rendered conversation looks like this:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of Finland?<|im_end|>
<|im_start|>assistant
<think>
</think>
The capital of Finland is Helsinki.<|im_end|>
<|im_end|>. This is the token that SFT taught the model to
emit. The pretraining terminator <eos> does not occur in the SFT data. Use <|im_end|> as the
stop token.<think> block. The template put a
<think>\n…\n</think>\n\n block at the start of the final assistant turn of every
conversation. Dolci Instruct has almost no reasoning traces (385 of 2,152,112 conversations have
</think> in an assistant message), so the block was nearly always empty. This is not a
reasoning model.import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
enable_thinking=False, # puts the empty <think> block in the prompt, as in training
return_dict=True,
return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
generation_config.json lists <|im_end|> (4) and <eos> (2) as stop tokens, so generate()
stops at the end of the assistant turn.
The model was trained on all 2,152,112 conversations of the train split of
allenai/Dolci-Instruct-SFT
(revision main). Most of the data is English. The dataset card lists the sources and their licenses. The dataset is
licensed under ODC-BY and is intended for research and educational use under Ai2's
Responsible Use Guidelines.
Before training, the pipeline removed each conversation that had no supervised assistant token in its first 32,768 tokens. It kept conversations where the 32,768-token limit cut a supervised span.
Full-parameter SFT with TRL SFTTrainer.
Loss masking. The loss covers only the assistant tokens that come after the last user message in
each conversation, including their closing <|im_end|>. System, user, and tool tokens, and all
assistant turns before the last user message, are context only. In a multi-turn conversation, only
the final assistant reply (or the chain of assistant turns after the last user message) is trained.
Packing. Conversations were packed with TRL's best-fit-decreasing (BFD) strategy into sequences of up to 32,768 tokens, one packed sequence per device per step. If a conversation was longer than 32,768 tokens, only its first 32,768 tokens were used.
| Hyperparameter | Value |
|---|---|
| Epochs | 2 |
| Maximum sequence length | 32,768 |
| Global batch | 32 packed sequences (1 per device × 32 devices × 1 accumulation step), up to 1,048,576 tokens per step |
| Optimizer | AdamW (DeepSpeed), β₁ = 0.9, β₂ = 0.95, ε = 1e-8 |
| Weight decay | 0.0 |
| Peak learning rate | 8e-5 |
| Schedule | Linear warmup for 3% of steps, then linear decay to 0 |
| Gradient clipping | 1.0 |
| Precision | BF16 |
| Parallelism | DeepSpeed ZeRO stage 2 |
| Memory and speed | Gradient checkpointing (non-reentrant), Liger kernels, FlashAttention-2, torch.compile (Inductor) |
| Seed | 42 |
| Optimizer steps | 2,774 |
| Training tokens | ~2.8B |
| Item | Value |
|---|---|
| Cluster | LUMI (CSC, Finland) |
| Accelerators | 4 nodes × 8 AMD MI250X GCDs = 32 GCDs |
| Software | ROCm 7.2.4, Python 3.12, PyTorch 2.9.1, TRL 1.7.0 (Singularity container) |
| Wall-clock time | 31 hours (about 990 GCD-hours) |
| Item | Value |
|---|---|
| Run name | oellm-9b-256k-theta64m-prelude-anneal300b-sft |
| Base model | birgermoell/oellm-9b-256k-theta64m-prelude-anneal300b @ b61957c3e22e6145ee7973f415c45047a0f9a108 |
| Tokenizer | openeurollm/tokenizer-256k @ 94bae05cf351c4feb435d6a73c2aa8edfd88ae04 (branch qwen3-tokens) |
| Dataset | allenai/Dolci-Instruct-SFT, split train @ main |
| Training code | OpenEuroLLM/post-training @ b81119e32a368cfe77f33753731ed7672918581e, chat template qwen3 |
The SFT data comes from the Olmo 3 release:
@misc{olmo2025olmo3,
title={Olmo 3},
author={Team Olmo},
year={2025},
eprint={2512.13961},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.13961},
}
Training ran on the LUMI supercomputer, hosted by CSC in Finland.