BigCodeBench Dataset Description Homepage: https://bigcode bench.github.io/ Repository: https://github.com/bigcode project/bigcodebench Paper: Link Point of Contact: contact@bigcode project.org terry.zhuo@monash.edu The dataset has 2 variants: 1. BigCodeBench Complete : Code Completion based on the structured docstrings . 1. BigCodeBench Instruct : Code Generation based on the NL oriented instructions . The overall statistics of the dataset are as follows: Complete Instruct Task 1140 1140 Avg. Test Cases 5.6 5.6 Avg. Coverage 99% 99% Avg. Prompt Char. 1112.5 663.2 Avg. Prompt Line 33.5 11.7 Avg. Prompt Char. (Code) 1112.5 124.0 Avg. Solution Char. 426.0 426.0 Avg. Solution Line 10.0 10.0 Avg. Solution Cyclomatic Complexity 3.1 3.1 The function calling (tool use) statistics of the dataset are as follows: Complete/Instruct Domain 7 Standard Library 77 3rd Party Library 62 Standard Function Call 281 3rd Party Function Call 116 Avg. Task Library 2.8 Avg. Task Fun Call 4.7 Library Combo 577 Function Call Combo 1045 Domain Combo 56 Changelog Release Description v0.1.0 Initial release of BigCodeBench Dataset Summary BigCodeBench is an easy to use benchmark which evaluates LLMs with practi…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy