CAR Bench Dataset CAR Bench is a benchmark for evaluating AI voice assistants in a realistic automotive (car) environment. It tests an agent's ability to correctly use vehicle control tools, handle disambiguation, and avoid hallucinations. Dataset Structure The dataset is organized into task configs and mock data configs : Tasks Each task defines a user persona, an instruction, the initial vehicle/environment context, and the ground truth sequence of tool call actions the assistant should perform. Config Description Train Test tasks base Standard tasks covering vehicle controls, navigation, calendar, etc. 50 50 tasks disambiguation Tasks requiring the agent to disambiguate parameters (internally via preferences or by asking the user) 30 26 tasks hallucination Tasks where certain tools/parameters are intentionally removed to test if the agent hallucinates 48 50 Task schema: Field Type Description task id string Unique task identifier persona string Description of the simulated user's personality and communication style calendar id string Reference to a calendar in the mock data instruction string The instruction given to the simulated user context init config string (JSON) Initial v…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy