NEvo

Community fork maintained by mzx. The original NEvo project and paper are by EPFL NeuroAI. This fork is an independent adaptation and is not affiliated with or endorsed by EPFL or the original authors. See the original model, project website, and paper.

🚧 Work in progress — this model is still being transferred from its main development repository, so the model card and API are subject to change.

NEvo — Neural-Guided Evolutionary Video Synthesis

🌐 Project website: nevo-project.epfl.ch · 📄 Paper: arXiv:2607.02317

NEvo is a self-contained Hugging Face custom Diffusers pipeline for neural-response-guided visual stimulus synthesis. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.

It orchestrates three frozen models. The models below are placeholders / defaults and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):

Gallery

Each clip is from the top results of a NEvo search targeting one visual region — the model discovers, from scratch, stimuli that drive that region's known selectivity.

Explore the full interactive gallery and 3D brain maps at nevo-project.epfl.ch.

Installation

Off-the-shelf — no install. Load NEvo as a custom Diffusers pipeline; the package and its bundled data are fetched from the Hub automatically (you only need the usual dependencies below):

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

Or install the package (for cleaner from stimulus_synthesis import ... imports / development):

conda create -n nevo python=3.10 -y
conda activate nevo
pip install "git+https://huggingface.co/epfl-neuroai/NEvo"
# or from a local clone:
#   git clone https://huggingface.co/epfl-neuroai/NEvo && pip install ./NEvo
# then:  from stimulus_synthesis import NevoPipeline; pipe = NevoPipeline.from_pretrained("epfl-neuroai/NEvo")

Runtime dependencies (either way): torch, diffusers, transformers, huggingface_hub, numpy, pillow, av (pytest for tests). No nilearn / atlas downloads — ROI masks are shipped as small precomputed data files.

Quickstart

Target a brain region by name — NEvo resolves its voxels and searches for a video predicted to drive it:

from diffusers import DiffusionPipeline

# fetches the pipeline (and package) from the Hub; model weights are pulled on first use
pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

out = pipe(roi="FFA", progress=True)                             # omit seed (default) -> different result each run; pass seed=<int> to reproduce (the seed used is in out.metadata["seed"])
print(out.best_prompt, out.best_score)

from stimulus_synthesis.media import save_video, video_to_t_c_h_w   # importable once the pipeline has loaded
save_video(video_to_t_c_h_w(out.best.video), "best_stimulus.mp4", fps=12)   # save the synthesized video (mp4 playback fps; default 12)
out.best.image.save("best_stimulus.png")                            # and the stage-1 best image (PIL)

This runs the two-stage search with the defaults — up to 400 image evaluations then 200 video evaluations (population 20), using the fast distilled-model defaults (1-step 512×512 SDXL-Turbo, 8-step 512×512 LTX). A run takes a few minutes and a good amount of GPU memory.

Faster run

For a quicker first result, shrink the search and the video:

out = pipe(
    roi="FFA",
    progress=True,
    image_max_evals=80,       # stage-1 (image) evaluation budget (default: 400)
    video_max_evals=40,       # stage-2 (video) evaluation budget (default: 200)
    population_size=8,        # GA population per generation (default: 20)
    image_batch_size=8,       # images generated per batch (default: 16)
    video_batch_size=4,       # videos generated per batch (default: 8)
    seed=0,                   # RNG seed, for reproducibility
    video_kwargs={            # merged over the fast defaults (8 steps / 25 frames / 512²); override any key
        "num_inference_steps": 8,   # denoising steps — the distilled LTX model needs only a few
        "num_frames": 25,           # clip length; LTX requires 8*k + 1 frames
        "height": 256, "width": 256,
    },
)

Enhanced search space. Selecting a region (roi=...) restricts the prompt search to the categories relevant to that region — a smaller space that converges faster. Pass enforce_general_search_space=True to search the full general space instead.

Available ROI tokens (comma-separated tokens are unioned):

• Named ROIs: FFA, PPA, MT, EBA, LOC, RSC, pSTS, aSTS, V1, V2, V3, V4 — optionally hemisphere-suffixed (FFA_lh, MT_rh).
• Searchlight regions: SL-<n> (both hemispheres), SL-<n>_lh, SL-<n>_rh (58 both / 28 lh / 30 rh).

from stimulus_synthesis.neuro import available_rois, searchlight_counts
available_rois()        # ['EBA','FFA','LOC','MT','PPA','RSC','V1','V2','V3','V4','aSTS','pSTS']
searchlight_counts()    # {'both': 58, 'lh': 28, 'rh': 30}

fsaverage5 only. The bundled ROI/searchlight masks are defined on the fsaverage5 cortical surface (20 484 vertices). Targeting a region by name therefore requires an encoder whose predict_fmri output lives in that same space — the default epfl-neuroai/vjepa2-encoder-basic. A custom encoder with a different output space can still be driven with explicit vector/indices targets, but not with the named-ROI helper.

Custom targets

Instead of a region name, pass raw voxel indices or a full target fMRI vector:

import numpy as np
from stimulus_synthesis import resolve_driving_voxels

mask = resolve_driving_voxels("FFA")                 # boolean mask, length 20484
out = pipe(target={"type": "indices", "indices": np.flatnonzero(mask).tolist()})

Target types & objectives

Search parameters (defaults)

Set in stimulus_synthesis_config.json:

Each stage runs a genetic search with population population_size (default 20) until it hits its evaluation budget — image_max_evals (default 400) and video_max_evals (default 200) generate→score passes. Image and video generation use fast distilled defaults out of the box (default_image_kwargs / default_video_kwargs); anything you pass as image_kwargs / video_kwargs is merged over them, so you only override the keys you care about.

Robust scoring

By default each candidate is scored with a single clean encoder pass. An optional robust mode — the mean over 4 augmented draws (random crop 0.8, Gaussian σ=0.1) via RobustTransformScorer — reduces sensitivity to encoder artifacts; turn it on by setting "enabled": true in default_score_transform.

Cache configuration

Model weights and outputs cache location resolves in priority order:

1. NEvo_CACHE_DIR — set it in a repo-root .env file (see .env.example) or the environment.
2. Otherwise the system/user-default HuggingFace cache (HF_HOME, else ~/.cache/huggingface) is used and left untouched.
3. Only if no default is resolvable, a repo-local cache/ is used.

cache/ and .env are git-ignored.

Batch runners

Two ROI-driven, two-stage (image-search → video-search) runners are included:

• run_roi_samples.py — genetic search per ROI/seed, scoring in-memory tensors; writes best_image.png / best_video.mp4 / scores.
• run_regional_asset_pilot.py — same search but exports every candidate to a deterministically-encoded file (PNG/MP4), hashes it (sha256), and scores the decoded file — producing provenance-tracked, reproducible published assets with manifests.

Both take --rois, --seeds, --image-evals / --video-evals, --encoder-model, --out-dir, etc., and default to the config's encoder and a cache-relative output directory.

Reproducibility

The pipeline is deterministic for a fixed seed/config: the shipped ROI masks reproduce the original atlas masks bit-for-bit, and a fixed-seed run reproduces prior scores exactly. Encoder scores are a target-matching signal, not ground-truth reconstruction quality.

Intended use & limitations

• Research use in visual neuroscience / brain-decoding. Outputs are predicted to drive a target region under a specific encoder — they are hypotheses to validate, not ground truth.
• Optimizing hard against a single encoder can exploit encoder artifacts; inspect images and use held-out validation.
• Requires a CUDA GPU with enough memory for the 13B video model; you must accept the license/access terms of the referenced upstream models.

Citation

If you use NEvo, please cite:

@article{tang2026nevo,
  title={NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity},
  author={Tang, Yingtian and Salehi, Sogand and Zhou, Ming and Zamir, Amir and Isik, Leyla and Schrimpf, Martin},
  journal={arXiv preprint arXiv:2607.02317},
  year={2026}
}

Project website: nevo-project.epfl.ch

Acknowledgements

Builds on BrainDiVE-style encoder-guided synthesis, vJEPA-2, SDXL-Turbo, and LTX-Video. ROI/searchlight definitions derive from an fsaverage-space group atlas (precomputed and bundled).

Intended Use, Out-of-Scope Use & Risks

NEvo is a research-only tool for computational visual neuroscience: it searches for stimuli predicted to drive a target brain region under a fixed image/video→fMRI encoder, and its outputs are unvalidated hypotheses, not scientific findings. It is not a medical device and must not be used for diagnosis, treatment, or to design, guide, or deliver any real brain stimulation (TMS, tDCS/tACS, DBS, intracranial, etc.) or in-vivo neuromodulation — it only optimizes a predicted fMRI response, not effects on a living brain, and any use involving human or animal subjects requires independent ethical review (e.g. IRB). Optimizing against a single encoder can exploit artifacts, and the generative models may produce biased or disturbing imagery; the model is provided as is, without warranty, for non-commercial use under CC BY-NC 4.0 (upstream models carry their own licenses).