Olmo-3-Instruct-imend-eos

Olmo-3-Instruct tokenizer (eos_token fixed to <|im_end|>)

Tokenizer-only fork of the Olmo 3 Instruct family (e.g. allenai/Olmo-3-7B-Instruct) — should work for any size sharing the same Instruct tokenizer/chat template. No model weights here — use this repo for the tokenizer/chat template only, alongside the original repo's weights.

What changed

eos_token is set to <|im_end|> (token id 100265) instead of the upstream default <|endoftext|> (token id 100257). Everything else (vocab, chat_template.jinja, special tokens) is untouched.

Why

Olmo 3's chat template only emits the literal eos_token after the final assistant turn when that turn is the last message in the list:

{%- if not loop.last -%}{{- '<|im_end|>' + '\n' -}}
{%- else -%}{{- eos_token -}}{%- endif -%}

Every other turn boundary closes with <|im_end|> unconditionally. Tooling that renders a partial trajectory (e.g. to find the templated prefix before the last assistant turn, without that turn present) never sees the real eos_token and errors — even though <|im_end|> (id 100265) is already registered as a valid stop token in the model's own generation_config.json (eos_token_id: [100265, 100257]), and is a real, official special token (special: true) in the model's own vocabulary. This fork just makes tokenizer.eos_token point at the one that actually appears at every turn boundary, sidestepping the conditional.