Dataset Card for MultiPL E Dataset Description Repository: https://github.com/nuprl/MultiPL E Paper: https://ieeexplore.ieee.org/abstract/document/10103177 Point of Contact: carolyn.anderson@wellesley.edu, mfeldman@oberlin.edu, a.guha@northeastern.edu Dataset Summary MultiPL E is a dataset for evaluating large language models for code generation that supports 22 programming languages. It takes the OpenAI HumanEval and the Mostly Basic Python Programs (MBPP) benchmarks and uses little compilers to translate them to other languages. It is easy to add support for new languages and benchmarks. The dataset is divided into several configurations named SRCDATA LANG , where SRCDATA is either "humaneval" or "mbpp" and LANG is one of the supported languages. We use the canonical file extension for each language to identify the language, e.g., "cpp" for C++, "lua" for Lua, "clj" for Clojure, and so on. Using MultiPL E MultiPL E is part of the [BigCode Code Generation LM Harness]. This is the easiest way to use MultiPL E. MultiPL E has its own evaluation framework that supports proprietary models, the prompt ablations, more source benchmarks, and more recently added programming languages. See…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy