FRAMES: Factuality, Retrieval, And reasoning MEasurement Set FRAMES is a comprehensive evaluation dataset designed to test the capabilities of Retrieval Augmented Generation (RAG) systems across factuality, retrieval accuracy, and reasoning. Our paper with details and experiments is available on arXiv: https://arxiv.org/abs/2409.12941. Dataset Overview 824 challenging multi hop questions requiring information from 2 15 Wikipedia articles Questions span diverse topics including history, sports, science, animals, health, etc. Each question is labeled with reasoning types: numerical, tabular, multiple constraints, temporal, and post processing Gold answers and relevant Wikipedia articles provided for each question Key Features Tests end to end RAG capabilities in a unified framework Requires integration of information from multiple sources Incorporates complex reasoning and temporal disambiguation Designed to be challenging for state of the art language models Usage This dataset can be used to: Evaluate RAG system performance Benchmark language model factuality and reasoning Develop and test multi hop retrieval strategies Baseline Results We provide baseline results using state of the…
Runs entirely in your browser via DuckDB-Wasm — this dataset's real data file is loaded once, then queried locally. Nothing is sent to a server.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy