mochi-1-preview-mix-nf4-small

Please change bf16 to fp16 in your code. I've just changed the checkpoint to fp16, which appears more coherent with mochi

This is a nf4 quantization of mochi-1-preview. It excludes specific layers that allow it to stay coherent:

bf16

<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/630406731dd5d3c62486140a/tnCaH4bMgvjYi6Wfjk3vb.mp4"></video>

nf4mix-small (this one):

<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/630406731dd5d3c62486140a/DqXbARydKh0ts-bgzSGrt.mp4"></video>

nf4:

<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/630406731dd5d3c62486140a/gVMUZNcAPxGSoiPRximAN.mp4"></video>

Diffusers main branch is required

pip install git+https://github.com/huggingface/diffusers

To use:

from diffusers import MochiPipeline, MochiTransformer3DModel
from diffusers.utils import export_to_video
import torch
transformer = MochiTransformer3DModel.from_pretrained("imnotednamode/mochi-1-preview-mix-nf4-small", torch_dtype=torch.float16)
pipe = MochiPipeline.from_pretrained("genmo/mochi-1-preview", revision="refs/pr/18", torch_dtype=torch.float16, transformer=transformer)
pipe.enable_model_cpu_offload()
pipe.enable_vae_tiling()
frames = pipe("A camera follows a squirrel running around on a tree branch", num_inference_steps=100, guidance_scale=4.5, height=480, width=848, num_frames=161).frames[0]
export_to_video(frames, "mochi.mp4", fps=15)

To reproduce:

from diffusers import MochiPipeline, MochiTransformer3DModel, BitsAndBytesConfig
import torch
quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_quant_type="nf4", llm_int8_skip_modules=["final_layer", "x_embedder.proj", "t_embedder", "pos_frequencies", "t5"])
# Please convert mochi to diffusers first
transformer = MochiTransformer3DModel.from_pretrained("genmo/mochi-1-preview", revision="refs/pr/18", subfolder="transformer", quantization_config=quantization_config, torch_dtype=torch.float16)
transformer.save_pretrained("mochi-1-preview-nf4")