NEvo

🚧 Work in progress — this model is still being transferred from its main development repository, so the model card and API are subject to change.

NEvo — Neural-Guided Evolutionary Video Synthesis

🌐 Project website: nevo-project.epfl.ch · 📄 Paper: arXiv:2607.02317

NEvo is a self-contained Hugging Face custom Diffusers pipeline for neural-response-guided visual stimulus synthesis. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.

It orchestrates three frozen models. The models below are placeholders / defaults and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):

Gallery

Each clip is from the top results of a NEvo search targeting one visual region — the model discovers, from scratch, stimuli that drive that region's known selectivity.

Explore the full interactive gallery and 3D brain maps at nevo-project.epfl.ch.

Installation

Off-the-shelf — no install. Load NEvo as a custom Diffusers pipeline; the package and its bundled data are fetched from the Hub automatically (you only need the usual dependencies below):

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

Or install the package (for cleaner from stimulus_synthesis import ... imports / development):

conda create -n nevo python=3.10 -y
conda activate nevo
pip install "git+https://huggingface.co/epfl-neuroai/NEvo"
# or from a local clone:
#   git clone https://huggingface.co/epfl-neuroai/NEvo && pip install ./NEvo
# then:  from stimulus_synthesis import NevoPipeline; pipe = NevoPipeline.from_pretrained("epfl-neuroai/NEvo")

Runtime dependencies (either way): torch, diffusers, transformers, huggingface_hub, numpy, pillow, av (pytest for tests). No nilearn / atlas downloads — ROI masks are shipped as small precomputed data files.

Quickstart

Target a brain region by name — NEvo resolves its voxels and searches for a video predicted to drive it:

from diffusers import DiffusionPipeline

# fetches the pipeline (and package) from the Hub; model weights are pulled on first use
pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

out = pipe(roi="FFA", progress=True)                             # omit seed (default) -> different result each run; pass seed=<int> to reproduce (the seed used is in out.metadata["seed"])
print(out.best_prompt, out.best_score)

from stimulus_synthesis.media import save_video, video_to_t_c_h_w   # importable once the pipeline has loaded
save_video(video_to_t_c_h_w(out.best.video), "best_stimulus.mp4", fps=12)   # save the synthesized video (mp4 playback fps; default 12)
out.best.image.save("best_stimulus.png")                            # and the stage-1 best image (PIL)

This runs the two-stage search with the defaults — up to 400 image evaluations then 200 video evaluations (population 20), using the fast distilled-model defaults (1-step 512×512 SDXL-Turbo, 8-step 512×512 LTX). A run takes a few minutes and a good amount of GPU memory.

Faster run

For a quicker first result, shrink the search and the video:

out = pipe(
    roi="FFA",
    progress=True,
    image_max_evals=80,       # stage-1 (image) evaluation budget (default: 400)
    video_max_evals=40,       # stage-2 (video) evaluation budget (default: 200)
    population_size=8,        # GA population per generation (default: 20)
    image_batch_size=8,       # images generated per batch (default: 16)
    video_batch_size=4,       # videos generated per batch (default: 8)
    seed=0,                   # RNG seed, for reproducibility
    video_kwargs={            # merged over the fast defaults (8 steps / 25 frames / 512²); override any key
        "num_inference_steps": 8,   # denoising steps — the distilled LTX model needs only a few
        "num_frames": 25,           # clip length; LTX requires 8*k + 1 frames
        "height": 256, "width": 256,
    },
)

Enhanced search space. Selecting a region (roi=...) restricts the prompt search to the categories relevant to that region — a smaller space that converges faster. Pass enforce_general_search_space=True to search the full general space instead.

Available ROI tokens (comma-separated tokens are unioned):

• Named ROIs: FFA, PPA, MT, EBA, LOC, RSC, pSTS, aSTS, V1, V2, V3, V4 — optionally hemisphere-suffixed (FFA_lh, MT_rh).
• Searchlight regions: SL-<n> (both hemispheres), SL-<n>_lh, SL-<n>_rh (58 both / 28 lh / 30 rh).

from stimulus_synthesis.neuro import available_rois, searchlight_counts
available_rois()        # ['EBA','FFA','LOC','MT','PPA','RSC','V1','V2','V3','V4','aSTS','pSTS']
searchlight_counts()    # {'both': 58, 'lh': 28, 'rh': 30}

fsaverage5 only. The bundled ROI/searchlight masks are defined on the fsaverage5 cortical surface (20 484 vertices). Targeting a region by name therefore requires an encoder whose predict_fmri output lives in that same space — the default epfl-neuroai/vjepa2-encoder-basic. A custom encoder with a different output space can still be driven with explicit vector/indices targets, but not with the named-ROI helper.

Custom targets

Instead of a region name, pass raw voxel indices or a full target fMRI vector:

import numpy as np
from stimulus_synthesis import resolve_driving_voxels

mask = resolve_driving_voxels("FFA")                 # boolean mask, length 20484
out = pipe(target={"type": "indices", "indices": np.flatnonzero(mask).tolist()})

Target types & objectives

Search parameters (defaults)

Set in stimulus_synthesis_config.json:

Each stage runs a genetic search with population population_size (default 20) until it hits its evaluation budget — image_max_evals (default 400) and video_max_evals (default 200) generate→score passes. Image and video generation use fast distilled defaults out of the box (default_image_kwargs / default_video_kwargs); anything you pass as image_kwargs / video_kwargs is merged over them, so you only override the keys you care about.

Robust scoring

By default each candidate is scored with a single clean encoder pass. An optional robust mode — the mean over 4 augmented draws (random crop 0.8, Gaussian σ=0.1) via RobustTransformScorer — reduces sensitivity to encoder artifacts; turn it on by setting "enabled": true in default_score_transform.

Cache configuration

Model weights and outputs cache location resolves in priority order:

1. NEvo_CACHE_DIR — set it in a repo-root .env file (see .env.example) or the environment.
2. Otherwise the system/user-default HuggingFace cache (HF_HOME, else ~/.cache/huggingface) is used and left untouched.
3. Only if no default is resolvable, a repo-local cache/ is used.

cache/ and .env are git-ignored.

Batch runners

Two ROI-driven, two-stage (image-search → video-search) runners are included:

• run_roi_samples.py — genetic search per ROI/seed, scoring in-memory tensors; writes best_image.png / best_video.mp4 / scores.
• run_regional_asset_pilot.py — same search but exports every candidate to a deterministically-encoded file (PNG/MP4), hashes it (sha256), and scores the decoded file — producing provenance-tracked, reproducible published assets with manifests.

Both take --rois, --seeds, --image-evals / --video-evals, --encoder-model, --out-dir, etc., and default to the config's encoder and a cache-relative output directory.

Reproducibility

The pipeline is deterministic for a fixed seed/config: the shipped ROI masks reproduce the original atlas masks bit-for-bit, and a fixed-seed run reproduces prior scores exactly. Encoder scores are a target-matching signal, not ground-truth reconstruction quality.

Intended use & limitations

• Research use in visual neuroscience / brain-decoding. Outputs are predicted to drive a target region under a specific encoder — they are hypotheses to validate, not ground truth.
• Optimizing hard against a single encoder can exploit encoder artifacts; inspect images and use held-out validation.
• Requires a CUDA GPU with enough memory for the 13B video model; you must accept the license/access terms of the referenced upstream models.

Citation

If you use NEvo, please cite:

@article{tang2026nevo,
  title={NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity},
  author={Tang, Yingtian and Salehi, Sogand and Zhou, Ming and Zamir, Amir and Isik, Leyla and Schrimpf, Martin},
  journal={arXiv preprint arXiv:2607.02317},
  year={2026}
}

Project website: nevo-project.epfl.ch

Acknowledgements

Builds on BrainDiVE-style encoder-guided synthesis, vJEPA-2, SDXL-Turbo, and LTX-Video. ROI/searchlight definitions derive from an fsaverage-space group atlas (precomputed and bundled).

Intended Use, Out-of-Scope Use & Risks

NEvo is a research-only tool for computational visual neuroscience: it searches for stimuli predicted to drive a target brain region under a fixed image/video→fMRI encoder, and its outputs are unvalidated hypotheses, not scientific findings. It is not a medical device and must not be used for diagnosis, treatment, or to design, guide, or deliver any real brain stimulation (TMS, tDCS/tACS, DBS, intracranial, etc.) or in-vivo neuromodulation — it only optimizes a predicted fMRI response, not effects on a living brain, and any use involving human or animal subjects requires independent ethical review (e.g. IRB). Optimizing against a single encoder can exploit artifacts, and the generative models may produce biased or disturbing imagery; the model is provided as is, without warranty, for non-commercial use under CC BY-NC 4.0 (upstream models carry their own licenses).