Amphora_NeuroText_legacy

Amphora NeuroText — Text & Audio → Brain ROI Activation

Three lightweight models trained on real naturalistic fMRI data that predict activation across 56 canonical brain ROIs from text or audio stimuli.

All models are trained in fsaverage5 + subcortical common space (28,444 voxels) and output predictions in the HCP MMP1.0 parcellation schema.

Model designed and built by Amphora, open source and free for commercial use.

---

Models

Val R = mean Pearson correlation across 56 ROIs on held-out real fMRI data.

---

Quick Start

from predict import predict_text, predict_audio, predict_combined, top_rois

# Text → ROI (qwen3 projector)
roi_map = predict_text(
    "watching a spider crawl toward me",
    "text2roi_projector.pt"
)
print(top_rois(roi_map, n=5))
# [('Amygdala_L', 0.54), ('AI', 0.53), ('dACC', 0.55), ...]

# Audio → ROI (whisper projector)
roi_map = predict_audio("speech_clip.wav", "text2roi_whisper.pt")

# Text + Audio → ROI (dual-tower)
roi_map = predict_combined("narration", "audio.wav", "text2roi_dual_v2.pt")

CLI

python predict.py "remembering your childhood home" --checkpoint text2roi_projector.pt
python predict.py "jazz improvisation" --audio clip.wav --checkpoint text2roi_whisper.pt
python predict.py "a story about loss" --audio story.wav --checkpoint text2roi_dual_v2.pt

---

Installation

pip install torch transformers librosa numpy
# For brain visualization:
pip install nibabel nilearn mne

---

56-ROI Schema

Visual:       V1 V2 V3 V4 V3A V3B LO1 LO2 MT MST V7 IPS1
              FFA-1 FFA-2 PPA RSC OFA EBA
Parietal:     IPS2 IPS3 IPS4 IPS5 SPL1 hIP1 hIP2 hIP3
Frontal:      dlPFC vlPFC OFC ACC mPFC FP1 FP2
Language:     IFG IFGorb STG STS MTG AG
Default mode: PCC mPFC_dmn LP_L LP_R HPC_L HPC_R
Salience:     AI dACC sgACC vmPFC
              Amygdala_L Amygdala_R
Subcortical:  Caudate_L Caudate_R Putamen_L Putamen_R Thalamus

---

Training Data

All three models are trained exclusively on real fMRI recordings — no synthetic data, no simulated activations.

Total: ~893k text-brain pairs (qwen3), ~373k audio-brain pairs (whisper). All preprocessed and projected to fsaverage5 + subcortical 28,444-voxel space.

---

Architecture

Text projector & Whisper projector (shared MLP class):

Linear(in_dim → 1024) → GELU → Dropout → LayerNorm
→ Linear(1024 → 512) → GELU → Dropout
→ Linear(512 → 56)

• Text projector: in_dim=2560 (Qwen3-Embedding-4B)
• Whisper projector: in_dim=1280 (Whisper-large-v3 encoder mean pool)

Dual-tower (modality-specialized heads):

whisper_tower(1280→512) → whisper_head → 6 auditory/salience ROIs
qwen3_tower(2560→512)   → qwen3_head  → 13 semantic/DMN ROIs
[wt ⊕ qt] → fusion_head             → 37 visual/parietal/subcortical ROIs

Training loss: pearson_r + 0.1 × MSE on real fMRI targets. Augmentation: 8× noisy brain target copies per batch (~6.4M effective pairs/epoch).

---

Visualization

See the included Colab notebook (text2roi_colab_v2.ipynb) for full visualization on a glass-brain and fsaverage5 surface using Nilearn/MNE.

---

Citation

If you use these models, please cite the underlying fMRI datasets:

• Narratives: Nastase et al. (2021). NeuroImage. doi:10.1016/j.neuroimage.2021.118023
• LPP: Dehghani et al. (2017). Language, Cognition and Neuroscience.
• HCP: Van Essen et al. (2013). NeuroImage. doi:10.1016/j.neuroimage.2013.05.041

---

Notes

• The whisper projector (R=0.413) is the strongest single model; it captures auditory cortex and salience network responses with high fidelity.
• The text projector (R=0.212) generalizes well to novel semantic content; best used for written stimuli, transcripts, or conceptual queries.
• The dual-tower (R=0.170) requires both modalities; useful when you have synchronized audio+transcript and want modality-partitioned predictions.
• Predictions are in z-score space relative to a per-subject resting baseline. Positive values = activation above baseline; negative = suppression.