COMPASS: COMPositional Analysis of SkillS COMPASS is a controlled evaluation benchmark for diagnosing compositional failure in vision language models (VLMs). It constructs captions from Visual Genome scene graphs with explicit object, attribute, and relation structure, enabling two targeted analyses: the compositional integration gap (cost of joint reasoning) and skill load (how each skill degrades as primitive counts increase). Dataset Summary VLMs consistently degrade as caption complexity grows, yet the sources of this degradation are poorly understood. COMPASS addresses this by: 1. Compositional Integration Gap — Comparing model performance on composed captions vs. their decomposed primitive counterparts under matched perturbations, directly quantifying the cost of joint reasoning. 2. Skill Load — Measuring how each skill (object detection, attribute binding, relation reasoning) is affected by increasing counts of each primitive type, separating self load (same primitive type) from cross load (other primitive types). Key finding: compositional degradation reflects multiple separable factors . Joint reasoning accounts for only part of the observed decline; each skill degrades pr…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy