AutoGaze Project Page Paper GitHub Models & Data & Benchmark Demo AutoGaze is a ultra light weight model that automatically removes redundant patches in a video before passing it to any Vision Transformer (ViT) or Multi modal Large Language Model (MLLM). Speficially, AutoGaze perceives each frame and autoregressively selects ("gazing") a minimal set of patches that can reconstruct the original video (i.e., non redundant patches) up to a reconstruction loss threhold provided by the user. AutoGaze can self decide when to stop gazing for each frame based on user's request on the acceptable maximum reconstruction loss. Empircally, AutoGaze can reduce tokens in ViTs/MLLMs by up to 100x, reducing their latency by up to 19x/10x. This enables efficiently scaling MLLMs to 4K resolution, 1K frame videos, improving performance on benchmarks such as VideoMME. Especially, it improves performance by 14% on HLVid, a high resolution long form video benchmark proposed in this work as well. This model is for research and development only. Quick Start: See our GitHub repo for instructions on how to use AutoGaze. License/Terms of Use: NVIDIA license (see LICENSE.md). The reference to the NVIDIA Licens…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy