Perception Encoder Audio Visual (PE AV) PE AV is a state of the art multimodal model that embeds audio, video, audio video, and text into a joint embedding space. The model enables powerful cross modal retrieval and understanding across audio, video, and text modalities. Model Description PE AV is trained using contrastive learning to align audio, video, and text representations in a shared embedding space. The model can encode: Audio only : Extract audio embeddings from audio waveforms Video only : Extract visual embeddings from video frames Audio Video : Extract joint audio visual embeddings Text : Extract text embeddings optimized for different modality pairs Model Variants We release 6 model checkpoints with varying sizes and capabilities: Model Avg Retrieval Video Frames used pe av small 16 frame 45.2 16 frames pe av base 16 frame 47.0 16 frames pe av large 16 frame 48.2 16 frames pe av small 48.1 all frames pe av base 50.2 all frames pe av large 51.6 all frames The 16 frame variants sample exactly 16 frames (evenly spaced apart) from each video, while the base variants support variable length videos. Quick Start The model is available in both transformers as well as perceptio…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy