OpenMythos-1.5b-transplant-shell-preview

OpenMythos Qwen Shell

OpenMythos Qwen Shell is an experimental compatibility checkpoint that loads Qwen2.5-1.5B-Instruct weights into an OpenMythos PyTorch shell.

This is not yet the final compressed/recurrent OpenMythos model. This checkpoint is the first successful donor-compatibility milestone: Qwen2.5-1.5B-Instruct was transplanted into the OpenMythos runtime and generated coherent text through the OpenMythos forward/generation path.

Summary

This model demonstrates that OpenMythos can host real open-weight transformer checkpoints without breaking generation.

The coherent checkpoint uses:

• Qwen2.5-1.5B-Instruct tokenizer
• Qwen token embeddings
• Qwen-compatible GQA attention
• Qwen Q/K/V projection biases
• Qwen-style RoPE behavior
• fp32 RMSNorm stabilization - otherwise breaks
• PyTorch SDPA attention path - potential for flash-attn but haven't tried
• Qwen MLP weights
• Qwen final normalization and LM head
• OpenMythos model shell and inference path

The successful coherent demo generated:

The vast blue sky stretches endlessly above, filled with fluffy white clouds that drift lazily across its canvas.

What This Model Is

This checkpoint is best described as:

Qwen2.5-1.5B-Instruct weights transplanted into an OpenMythos-compatible shell.

The coherent version uses a Qwen-compatible layer layout:

Qwen layers 0-13    -> OpenMythos prelude
Qwen layer 14       -> OpenMythos recurrent slot, patched to act like a normal Qwen-style block
Qwen layers 15-27   -> OpenMythos coda
Qwen embedding/head -> OpenMythos embedding/head path


This checkpoint is not the final compressed recurrent OpenMythos model.
The true compressed/recurrent OpenMythos student is a separate ongoing experiment that requires distillation.

Installation

pip install -U torch transformers accelerate safetensors sentencepiece

python infer_saved_openmythos_qwen.py \
  --ckpt_dir qwen2_5_1_5b_openmythos_full_shell \
  --prompt "Write a python script of the GPT architecture" \
  --max_new_tokens 100


Simple generation - 

[prompt] What can you tell me about Qwen?

[output]
Qwen is an AI language model created by Alibaba Cloud.
It is designed to be a general-purpose language model that can be used for a wide range of tasks, including text generation, question answering, and more.
Qwen is trained on a large corpus of text data, and it uses advanced natural language processing techniques to understand and generate human-like language.
It is available as a cloud service and can be used by developers and businesses to build their own AI-powered applications.

Limitations

This model is experimental. It behaves like qwen2.5-1.5b simply due to the recurrent layer not looping thus acting as the regular layer.

Attribution

This checkpoint is based on Qwen2.5-1.5B-Instruct by the Qwen team.

Base model:

Qwen/Qwen2.5-1.5B-Instruct

This repository adds an experimental OpenMythos-compatible shell, compatibility patches, transplant logic, and inference scripts.

License

This repository is released under Apache-2.0, following the upstream Qwen2.5-1.5B-Instruct license.

Please follow the terms of the upstream Qwen model license and include appropriate attribution when redistributing or modifying this work.