github xCodeEval xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval We introduce xCodeEval , the largest executable multilingual multitask benchmark to date consisting of 25 M document level coding examples from about 7.5 K unique problems covering up to 17 programming languages with execution level parallelism. It features a total of seven tasks involving code understanding, generation, translation and retrieval, and it employs an execution based evaluation. We develop a test case based multilingual code execution engine, ExecEval that supports all the programming languages in xCodeEval . We also propose a novel data splitting and a data selection schema for balancing data distributions over multiple attributes based on geometric mean and graph theoretic principle. This repository contains the sample code and data link for xCodeEval paper. Data Download Currently this repository supports huggingface load dataset() api. Follow the following example to load dataset for individual examples. Hf large data download tricks. If you are facing long delay with data processing, add a ignore verifications=True . If you are…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy