ss d128 f1 Weight sparse transformer trained with the procedure from Gao et al. (2025). Model Details Layers : 2 Model Dimension : 128 Context Length : 512 Head Dimension : 16 Vocabulary Size : 4096 Sparsity Weight Sparsity : False Target L0 Fraction : 1 Activation Sparsity : False Training Dataset : SimpleStories/SimpleStories Tokenizer : SimpleStories/SimpleStories 1.25M Total Tokens : 2,000,000,000 Training Run W&B Run : https://wandb.ai/training saes/my sparsity/runs/sdwteyxx Usage
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy