YuE2-3B-Diffusers

YuE2-3B Diffusers

<audio controls src="https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/yue2/city_lights.mp3"></audio>

A 69-second song generated with the code below, seed 831001.

Diffusers version of m-a-p/YuE2-3B and m-a-p/YuE2-Vae: the transformer, the VAE and the tokenizer. Run it with the blocks in OzzyGT/YuE2-Modular.

Note: This model requires tiktoken and the example uses soundfile, install them with pip install tiktoken soundfile.

Licensing

The weights are CC BY-NC 4.0 (non-commercial), from m-a-p; see WEIGHTS_LICENSE. The code is Apache-2.0, adapted from multimodal-art-projection/YuE; see LICENSE.

Sample song

The song above was generated with the following code:

import soundfile as sf
import torch
from diffusers import ModularPipelineBlocks


blocks = ModularPipelineBlocks.from_pretrained(  # load the blocks first to avoid warnings
    "OzzyGT/YuE2-Modular",
    trust_remote_code=True,
    components_repo="OzzyGT/YuE2-3B-Diffusers",
    trust_components_code=True,
)
pipe = blocks.init_pipeline()
pipe.load_components(dtype={"transformer": torch.bfloat16, "vae": torch.float32})
pipe.to("cuda")

style = "English, warm piano pop, expressive female voice, acoustic piano, rounded bass and light drums, lyrical memorable melody, unhurried phrasing, 88 BPM"
lyrics = """[Verse]
Neon fades along the lane
Footsteps keep the time of rain
Fold the night and leave it here
Morning has a sky to clear

[Chorus]
Let the day come into view
Every road begins with you
Hold a little room for light
We will sing beyond the night"""

state = pipe(style=style, lyrics=lyrics, seed=831001, use_cuda_graph=True)
sf.write("song.flac", state.get("audios")[0].T.numpy(), state.get("sample_rate"), subtype="PCM_24")