Can I Trust Your Answer? Visually Grounded Video Question Answering Introduction We study visually grounded VideoQA by forcing vision language models (VLMs) to answer questions and simultaneously ground the relevant video moments as visual evidences. We show that this task is easy for human yet is extremely challenging for existing VLMs, revealing that the strong QA performance of these models may largely due to short cut learning (e.g., language priors and spurious vision text correlations) versus faithful multimodal reasoning. By defining grounded VQA, we hope to discourage such short cut learning and spark more interpretable and trustworthy techniques. This repository holds our data and code to facilitate the study. Environment Assume you have installed Anaconda, please do the following to setup the environment: Preparation Please create a data folder outside this repo, so you have two folders in your workspace 'workspace/data/' and 'workspace/NExT GQA/'. Please download the related video feature or raw videos . Extract the feature into . If you download the raw videos, you need to decode each video at 6fps and then extract the frame feature of CLIP via the script provided in .…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy