UI Vision: A Desktop centric GUI Benchmark for Visual Perception and Interaction Introduction Autonomous agents that navigate Graphical User Interfaces (GUIs) to automate tasks like document editing and file management can greatly enhance computer workflows. While existing research focuses on online settings, desktop environments, critical for many professional and everyday tasks, remain underexplored due to data collection challenges and licensing issues. We introduce UI Vision, the first comprehensive, license permissive benchmark for offline, fine grained evaluation of computer use agents in real world desktop environments. Unlike online benchmarks, UI Vision provides: (i) dense, high quality annotations of human demonstrations, including bounding boxes, UI labels, and action trajectories (clicks, drags, and keyboard inputs) across 83 software applications, and (ii) three fine to coarse grained tasks Element Grounding, Layout Grounding, and Action Prediction with well defined metrics to rigorously evaluate agents' performance in desktop environments. Our evaluation reveals critical limitations in state of the art models like UI TARS 72B, including issues with understanding profe…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy