TRIBE v2 A Foundation Model of Vision, Audition, and Language for In Silico Neuroscience 📄 Paper ▶️ Demo 🤗 Weights TRIBE v2 is a deep multimodal brain encoding model that predicts fMRI brain responses to naturalistic stimuli (video, audio, text). It combines state of the art feature extractors — LLaMA 3.2 (text), V JEPA2 (video), and Wav2Vec BERT (audio) — into a unified Transformer architecture that maps multimodal representations onto the cortical surface. Quick start Load a pretrained model from HuggingFace and predict brain responses to a video: Predictions are for the "average" subject (see paper for details) and live on the fsaverage5 cortical mesh (~20k vertices). You can also pass text path or audio path to model.get events dataframe — text is automatically converted to speech and transcribed to obtain word level timings. For a full walkthrough with brain visualizations, see the Colab demo notebook. Installation Basic (inference only): With brain visualization : With training dependencies (PyTorch Lightning, W&B, etc.): Training a model from scratch 1. Set environment variables Configure data/output paths and Slurm partition (or edit tribev2/grids/defaults.py directly): 2…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy