OpenAI MRCR: Long context multiple needle in a haystack benchmark OpenAI MRCR (Multi round co reference resolution) is a long context dataset for benchmarking an LLM's ability to distinguish between multiple needles hidden in context. This eval is inspired by the MRCR eval first introduced by Gemini (https://arxiv.org/pdf/2409.12640v2). OpenAI MRCR expands the tasks's difficulty and provides opensource data for reproducing results. The task is as follows: The model is given a long, multi turn, synthetically generated conversation between user and model where the user asks for a piece of writing about a topic, e.g. "write a poem about tapirs" or "write a blog post about rocks". Hidden in this conversation are 2, 4, or 8 identical asks, and the model is ultimately prompted to return the i th instance of one of those asks. For example, "Return the 2nd poem about tapirs". Example conversation for 2 needle case: This eval is challenging because: The needles are selected from the same distribution as the distractors. All assistant responses are generated by gpt4o, so the needle blends in with the haystack. The model must distinguish order amongst the needles. The more needles, the harder…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy