Model TL;DR CLAP is to audio what CLIP is to image. This is an improved CLAP checkpoint, specifically trained on music. Description CLAP (Contrastive Language Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score. Usage You can use this model for zero shot audio classification or extracting audio and/or textual features. Uses Perform zero shot audio classification Using pipeline Run the model: You can also get the audio and text embeddings using ClapModel Run the model on CPU: Run the model on GPU: Citation If you are using this model for your work, please consider citing the original paper:
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy