SeaOtter-0.5-Beta

This is an early test checkpoint for generating piano music in MIDI format from natural language prompts. It takes text prompts describing tempo, mood, and key, then generates symbolic REMI tokens that convert into standard .mid files.

---

How It Was Trained

• Base Model: unsloth/Qwen2.5-3B-Instruct-bnb-4bit
• Fine-Tuning Method: LoRA (Rank 16, Alpha 16) targeting all linear attention layers (q, k, v, o, gate, up, down)
• Framework: Unsloth with TRL SFTTrainer
• Data Processing: MidiCaps audio captions paired with MIDI tracks tokenized into REMI format using MidiTok
• Prompt Format: ChatML format with loss calculated only on assistant token responses
• Length & Training Duration: 2048 sequence length trained for 1 full epoch with a learning rate of 1e-4 and cosine decay

---

Current State & Limitations (v0.1 Beta)

• Single Note Focus: After 1 epoch, the model mainly writes single note lines and rolling arpeggio patterns. It does not reliably play 3 or 4 note stacked chords at the same position yet.
• Prompt Sensitivity: Works best when given specific musical parameters (BPM, key signature, mood) rather than broad single-word requests.
• Vocabulary: Strictly outputs REMI tokens compatible with the included miditok_config.json.

---

What Is Coming Next (v1.1 Plan)

• Multi-Epoch Training: Training for 3 full epochs to give the weights enough passes to master complex chord harmony and rhythm variation.
• MAESTRO Dataset: Moving to over 200 hours of expressive solo piano performances from the MAESTRO dataset.
• Native Chord Tokens: Building the new tokenizer with use_chords=True so the model outputs full chords as single dedicated tokens.

---

How to Run Inference

import os
import torch
from unsloth import FastLanguageModel
from miditok import REMI
from miditok.classes import TokSequence
from symusic import Score

model_path = "eikovo/piano-midi-beta"
config_path = "miditok_config.json"

midi_tokenizer = REMI(params=config_path)

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_path,
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

prompt = "A soft, expressive piano piece in A minor at 80 BPM."
messages = [
    {"role": "user", "content": f"Generate piano MIDI tokens matching this description: {prompt}"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

with torch.inference_mode():
    outputs = model.generate(
        input_ids=inputs,
        max_new_tokens=1400,
        min_new_tokens=128,
        temperature=0.92,
        top_k=40,
        top_p=0.90,
        repetition_penalty=1.15,
        do_sample=True,
        use_cache=True,
        pad_token_id=tokenizer.eos_token_id
    )

raw_output = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
raw_tokens = [t.strip() for t in raw_output.split() if t.strip()]
valid_tokens = [t for t in raw_tokens if t in midi_tokenizer.vocab]

if not valid_tokens[0].startswith("Bar_") and not valid_tokens[0].startswith("Position_"):
    valid_tokens = ["Bar_None", "Position_0"] + valid_tokens

seq = TokSequence(tokens=valid_tokens)
midi_tokenizer.complete_sequence(seq)
score = midi_tokenizer.decode([seq])

score = score.resample(480)
score.dump_midi("output.mid")

Credits Eikovo for the: training, programming, testing, refining Discord: Eikovo