SimpleStories Model Family The SimpleStories models are a tiny model family created for interpretability research, trained on the SimpleStories dataset. Usage Model Variants Model Name n params n layers d model n heads n ctx d vocab SimpleStories 35M 35 million 12 512 8 512 4096 SimpleStories 30M 30 million 10 512 8 512 4096 SimpleStories 11M 11 million 6 384 6 512 4096 SimpleStories 5M 5 million 6 256 4 512 4096 SimpleStories 1.25M 1.25 million 4 128 4 512 4096 Performance Comparison Model evaluated generation quality metrics: Tokenizer We use a custom WordPiece tokenizer with a small vocabulary size of 4096. We conducted morphological analysis and coverage gain analysis on the dataset to build a small tokenizer without compromising on the quality of generation. Dataset The SimpleStories dataset is a collection of short stories generated by state of the art language models. It features: Story annotation with high level concepts: theme, topic, style, etc. Higher semantic and syntactic diversity through seeded story generation Generated by 2024 models Several NLP metrics pre computed to aid filtering ASCII only guarantee for the English dataset Read the dataset paper on arXiv. Train…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy