tinyMMLU Welcome to tinyMMLU! This dataset serves as a concise version of the MMLU dataset, offering a subset of 100 data points selected from the original compilation. tinyMMLU is designed to enable users to efficiently estimate the performance of a large language model (LLM) with reduced dataset size, saving computational resources while maintaining the essence of the MMLU evaluation. Features Compact Dataset: With only 100 data points, tinyMMLU provides a swift and efficient way to evaluate your LLM's performance against a benchmark set, maintaining the essence of the original MMLU dataset. Compatibility: tinyMMLU is compatible with evaluation using the lm evaluation harness, but can also be integrated into your custom pipeline. See below for more details. Model Evaluation With lm eval harness Users looking to evaluate a new model with tinyMMLU can use the lm evaluation harness (v0.4.1 or later). To do so, you can directly run your evaluation harness with tasks=tinyMMLU but without the num fewshot argument: LM eval harness will directly output the best accuracy estimator (IRT++), without any additional work required. Without lm eval harness tinyMMLU can be integrated into any ot…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy