license: gpl 2.0 Collinear Scaling Models Checkpoint repository for scaling law experiments comparing collinear (CO) and non collinear (NC) experimental designs. Directory Structure {dataset}/{design}/N {param count}/ Dataset : wikipedia , pes2o , cosmopedia , redpajama , c4 (plus fp16 and bigtpp variants) Design : colinear or non colinear N : Model parameter count (one of 14 canonical sizes from ~5M to ~70M) Experimental Designs Collinear (CO): Models are trained along a line in (N, D) space where D = TPP × N for varying TPP (tokens per parameter) values. A single model size N is swept across many TPP values. Non collinear (NC): Models are trained on a grid over (N, D) space ( NxD GRID ), varying both N and D independently. Holdout Sets Some checkpoints include HOLDOUT in the filename. These were held out from scaling law fitting and are used to evaluate extrapolation / interpolation accuracy of fitted scaling laws. Both CO and NC designs have holdout checkpoints: COLINEAR HOLDOUT → collinear holdout (held out TPP values) HOLDOUT (without COLINEAR ) → non collinear holdout (held out (N, D) pairs) Filename Convention {PREFIX}{DESIGN}N{approx size}[TPP{val}]D{tokens} {dataset} m{exa…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy