my-cool-model

GEN3C Setup Bundle (v2)

Updated based on the actual repository code. Key facts from the source:

• GEN3C runs on Linux only. Cosmos/Apex/transformer-engine require Linux. On Windows you must use WSL2 (Ubuntu 22.04 or 24.04 recommended).
• Default output resolution is 704 x 1280. Pass --height/--width to change it; the model is most tested at 704x1280.
• Frame count must satisfy (N - 1) % 120 == 0 (i.e. 121, 241, 361, 481, ...). The README's "121*N - 1" wording is wrong; the code is right.
• The conda env name is fixed at cosmos-predict1 by the YAML file.
• Custom trajectory IS now supported via the included monkey-patch module (gen3c_custom_trajectory.py). No GEN3C source modification needed.

Files

Workflow

1. One-time install (1-3 hours, dominated by Apex compile and checkpoint download)

export HF_TOKEN=hf_your_token_here
chmod +x setup_gen3c.sh
./setup_gen3c.sh

This follows the official INSTALL.md exactly:

• Clones the repo with submodules
• Creates cosmos-predict1 conda env from the bundled YAML (Python 3.10, CUDA 12.4, gcc 12.4)
• Installs pip requirements
• Patches transformer-engine symlinks
• Installs transformer-engine 1.12.0
• Builds NVIDIA Apex from source (this is the slow step)
• Installs MoGe for depth estimation
• Authenticates Hugging Face
• Downloads the GEN3C-Cosmos-7B checkpoints
• Runs the bundled test_environment.py

2. Sanity check the install with the canned example

conda activate cosmos-predict1
cd ~/Projects/GEN3C
export CUDA_HOME=$CONDA_PREFIX
export PYTHONPATH=$(pwd)
python cosmos_predict1/diffusion/inference/gen3c_single_image.py \
    --checkpoint_dir checkpoints \
    --input_image_path assets/diffusion/000000.png \
    --video_save_name test_install \
    --guidance 1 \
    --foreground_masking

Verify the output video looks reasonable before doing anything custom.

3. Run a preset trajectory on your image

python /path/to/run_gen3c.py \
    --gen3c_dir $(pwd) \
    --image /path/to/Surius.png \
    --output_name surius_zoom_out \
    --preset_trajectory zoom_out \
    --movement_distance 0.5 \
    --num_video_frames 121

This confirms GEN3C handles your specific image. If it works, move on.

4. Export your Blender camera

In Blender (Scripting workspace):

1. Load the .blend with the 3DE-driven camera animation
2. Set the camera as the active scene camera
3. Open blender_export_camera.py, run it
4. A camera_trajectory.json lands next to your .blend file

The JSON contains OpenCV-convention world-to-camera 4x4 matrices, which match GEN3C's internal format.

5. Run with your custom trajectory

python /path/to/run_gen3c.py \
    --gen3c_dir $(pwd) \
    --image /path/to/Surius.png \
    --output_name surius_blender_cam \
    --trajectory_json /path/to/camera_trajectory.json \
    --num_video_frames 121

The wrapper:

1. Reads the JSON for metadata
2. Snaps the frame count to GEN3C's required 120*N + 1 pattern
3. Sets up a monkey-patch via gen3c_custom_trajectory.py
4. Runs gen3c_single_image.py with --trajectory none, which the patch intercepts to inject the Blender-derived w2c matrices

How the custom trajectory injection works

GEN3C's pipeline:

1. gen3c_single_image.py calls MoGe to predict depth from the input image
2. Builds a 3D point cloud cache anchored at world-to-camera = identity
3. Calls generate_camera_trajectory() to get a sequence of w2c matrices
4. Renders the cache from each new camera position
5. Diffusion model fills in occluded regions and refines

gen3c_custom_trajectory.py replaces step 3 with:

• Load Blender w2c matrices from JSON
• Resample to match GEN3C's required frame count
• Re-anchor so frame 0 is identity (matching MoGe's coordinate system)
• Return as a (1, N, 4, 4) tensor in the format GEN3C expects

No GEN3C source files are modified, so upstream pulls still work.

Resolution and your hero shot

GEN3C generates at 704 x 1280 by default. For 1920 x 1080 delivery:

• Option A: Pass --height 1080 --width 1920 and hope the model behaves at non-tested resolutions. May produce artifacts.
• Option B (recommended): Generate at 704 x 1280, then upscale to 1920 x 1080 with FlashVSR, Topaz Video AI, RTX Video Super Resolution, or tile diffusion at low denoise.

The upscaling pass is also where you'd add the "high-frequency detail" that AI video models naturally lack.

VRAM expectations

GEN3C's README says max observed inference memory is ~43 GB with full offloading. For a Blackwell RTX PRO 6000:

• Unsliced (~96 GB): runs comfortably, no --offload_all needed
• 2x 48 GB vGPU slice: requires --offload_all, will be slow
• Smaller slices: won't fit

If you hit OOM, add --offload_all to the run_gen3c.py command.

Caveats and known unknowns

• I have not run this end-to-end. The patch module is based on the code in cosmos_predict1/diffusion/inference/camera_utils.py and the call site in gen3c_single_image.py. First run might surface issues with coordinate conventions, intrinsics scale, or depth-to-trajectory alignment that need tuning.
• If the Blender trajectory is re-anchored to identity at frame 0, the first frame of your output should look almost identical to your input image (because MoGe and the cache see the same view). If it doesn't, the coordinate conversion in blender_export_camera.py may need adjustment.
• The intrinsics from your Blender JSON are currently not used; MoGe's predicted intrinsics are used instead because GEN3C's pipeline tunes depth and intrinsics together. If you need to override that, edit the patch module to substitute the JSON intrinsics.

Troubleshooting

• Apex compilation fails: check your gcc version matches the env YAML (gcc 12.4). On WSL2 with Ubuntu 22.04 you may need to install build tools: sudo apt install build-essential.
• Out of memory during Apex compile: Apex uses lots of RAM during C++/CUDA build. Close other apps, or limit parallel jobs by setting MAX_JOBS=4 before the pip install.
• HuggingFace 401 on checkpoint download: your token may need access to gated NVIDIA models. Check the GEN3C-Cosmos-7B model card on HuggingFace and accept any license agreements there.
• "no kernel image available" at runtime: Blackwell may need PyTorch built with sm_90 or newer kernels. If the bundled PyTorch doesn't support it, you may need a nightly PyTorch build.
• First frame looks wrong / scene flipped: the OpenCV conversion in blender_export_camera.py may need adjustment. Try removing the _rebase_to_first_frame call in the patch module to see raw trajectory behavior.