Orpheus-Music-Transformer

Orpheus Music Transformer

SOTA 8k multi-instrumental music transformer trained on 2.31M+ high-quality MIDIs

---

Abstract

Project Los Angeles is very proud to present Orpheus Music Transformer, an efficient, SOTA transformer model for long-form, multi-instrumental music generation. At its core lies highly optimized 748M-parameter autoregressive transformer equipped with Rotary Positional Embeddings (RoPE) and Flash Attention, enabling sequence lengths up to 8 k tokens—sufficient to capture extended musical structures. Trained for three epochs on 2.31 million high-quality MIDI tracks from the Godzilla dataset, our model employs a compact 3-token-per-note and 7-token-per-tri-chord encoding, plus a novel duration-and-velocity-last ordering to enhance expressivity. It leverage PyTorch’s bfloat16 precision and memory-efficient sparse-dense products for accelerated inference on CUDA, and provide a top-p sampling filter with adjustable temperature.

The Gradio interface empowers users to upload seed MIDI files or generate from scratch, tune prime/generation token counts, control randomness (temperature, top-p), and optionally append drums or natural “outro” tokens. Generated outputs appear in ten parallel batches with synchronized audio previews and piano-roll plots. Users can iteratively add or remove entire batches to sculpt a final composition, which is rendered back into MIDI and audio via an integrated SoundFont pipeline. Our release demonstrates a seamless blend of state-of-the-art model performance, efficient MIDI tokenization, and user-centric design, fostering rapid exploration of algorithmic composition.

---

Models

Presented are twelve models:

Base models

Orpheus Music Transformer Large (748) Model

This is a base model that is capable of music generation/continuation and notes/drums inpainting

Orpheus Music Transformer Medium (479M) Model

This is a base model that is capable of music generation/continuation and notes/drums inpainting

Orpheus Bridge Music Transformer Model

This is an auxiliary model that is capable of seamless bridge inpainting/infilling in any music composition

Fine-tuned base models

Orpheus Karaoke Music Transformer Model

This is a fine-tuned version of the Orpheus base model that is capable of generating Karaoke lyrics template for any MIDI

Orpheus LRNO Segments Music Transformer Model

This is a fine-tuned version of the Orpheus base model that is capable of music segmentation and music segments generation

Orpheus Loops Music Transformer Model

This is a fine-tuned version of the Orpheus base model that is capable of generating multi-instrumental loops

Orpheus Children Songs Music Transformer Model

This is a fine-tuned version of the Orpheus base model that is great for working with children songs

Orpheus Mono Melodies Transformer Model

This is a fine-tuned version of the Orpheus base model that is great for working with compositions that have a monophonic melody

Aux models

Orpheus Music Transformer Classifier Model

This is a music classification model that is was trained to distinguish between human music and music generated by Orpheus base model

---

Live Hugging Face spaces demos

Orpheus Music Transformer

If you enjoyed any of the Orpheus Music Transformer demos, please star and duplicate. It helps a lot! 🤗

---

Inference notebooks

[NEW & SOTA] Orpheus Auto-Continuations Generator

Orpheus Drums Transformer

---

Training dataset code

Models were trained on select HQ MIDIs from Godzilla MIDI Dataset

Please check out Orpheus Taining Dataset Maker notebook for details

---

Models training code

Please check out Orpheus Music Transformer Maker code/notebook for details

---

Project Los Angeles

Tegridy Code 2026