GSM Symbolic This project accompanies the research paper, GSM Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. Getting Started In our first release, we provide data for GSM Symbolic, GSM Symbolic P1, and GSM Symbolic P2 variants. For each variant, we have released both templates (available on Github), and a sample of generated data that can be used for evaluation. To load the data, you can use the following code. Note that in terms of difficulty, GSM Symbolic P2 is more difficult than GSM Symbolic P1, and GSM Symbolic P1 is more difficult than the default GSM Symbolic. Generated Data Format Each document of a generated dataset (e.g., generated data/GSM symbolic.jsonl ) contains these fields: id & instance : The id is the ID of the template/question, while instance is the ID of the generated sample. For example, the first template will have id=0 , and if we generate 50 instances from that template, the instance will range from 0 to 49. Rather than creating 50 different datasets, we create a single dataset file, collect the model's responses for all the questions, and then group the answers by the instance ID to evaluate model performance ac…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy