lmarena ai/p2l 0.5b grk 01112025 Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user and prompt specific variations in model performance. To address this, we propose Prompt to Leaderboard (P2L), a method that produces leaderboards specific to a prompt. The core idea is to train an LLM taking natural language prompts as input to output a vector of coefficients which are then used to predict the human preference vote. The resulting prompt dependent leaderboards allow for unsupervised task specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Paper : Prompt to Leaderboard Code : lmarena/p2l This particular P2L model has a Grounded Rao Kupper regression head, which we define below: Let $$ Y\in \{\mathsf{A}, \mathsf{B}, \mathsf{tie}, \mathsf{bad}\} $$ and for the sake of notational convenience, let $$ \theta^ (z) = \big(\beta^ (z), \eta^ (z)\big);…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy