Dataset Card for WMDP The Weapons of Mass Destruction Proxy (WMDP) benchmark is a dataset of multiple choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. See our paper, website, and GitHub for more details! We implemented the WMDP evaluation in the lm evaluation harness framework to maintain consistency. The datasets are also available at the following mirrors with password wmdpmcqs : 1, 2 Update 2024 04 23: the WMDP multiple choice questions were modified due to issues with data formatting and unicode encoding. Some questions in WMDP Cyber were also removed for being excessively long, which makes evaluation with a fixed batch size challenging. Some questions in WMDP Bio were also removed for insufficient dual use potential (h/t folks from Google DeepMind and OpenAI). The modified version is now uploaded on all mirrors; please re download the dataset. Thanks! Update 2024 03 08: the WMDP Cyber dataset was slightly modified due to issues with cho…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy