MVBench Dataset Description Repository: MVBench Paper: 2311.17005 Point of Contact: mailto:kunchang li Important Update [18/10/2024] Due to NTU RGB+D License, 320 videos from NTU RGB+D need to be downloaded manually. Please visit ROSE Lab to access the data. We also provide a list of the 320 videos used in MVBench for your reference. We introduce a novel static to dynamic method for defining temporal related tasks. By converting static tasks into dynamic ones, we facilitate systematic generation of video tasks necessitating a wide range of temporal abilities, from perception to cognition. Guided by task definitions, we then automatically transform public video annotations into multiple choice QA for task evaluation. This unique paradigm enables efficient creation of MVBench with minimal manual intervention while ensuring evaluation fairness through ground truth video annotations and avoiding biased LLM scoring. The 20 temporal task examples are as follows. Evaluation An evaluation example is provided in mvbench.ipynb. Please follow the pipeline to prepare the evaluation code for various MLLMs. Preprocess : We preserve the raw video (high resolution, long duration, etc.) along with…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy