CoNeTTE model for Audio Captioning CoNeTTE is an audio captioning system, which generate a short textual description of the sound events in any audio file. The architecture and training are explained in the corresponding paper. The model has been developped by me (Étienne Labbé) during my PhD. Installation Usage with python The model can also accept several audio files at the same time (list[str]), or a list of pre loaded audio files (list[Tensor]). In this second case you also need to provide the sampling rate of this files: The model can also produces different captions using a Task Embedding input which indicates the dataset caption style. The default task is "clotho". Usage with command line Simply use the command conette predict with audio PATH1 PATH2 ... option. You can also export results to a CSV file using csv export PATH . Performance Test data SPIDEr (%) SPIDEr FL (%) FENSE (%) Vocab Outputs Scores AC test 44.14 43.98 60.81 309 Link Link CL eval 30.97 30.87 51.72 636 Link Link This model checkpoint has been trained for the Clotho dataset, but it can also reach a good performance on AudioCaps with the "audiocaps" task. Limitations The model expected audio sampled at 32 kH…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy