This is a first version of recreating roneneldan/TinyStories 1M but using Llama architecture. Full training process is included in the notebook train.ipynb. Recreating it as simple as downloading TinyStoriesV2 GPT4 train.txt and TinyStoriesV2 GPT4 valid.txt in the same folder with the notebook and running the cells. Validation content is not used by the script so you put anythin in Backup directory has a script do backup that I used to copy weights from remote machine to local. Weight are generated too quickly, so by the time script copied weihgt N+1 This is extremely PoC version. Training truncates stories that are longer than context size and doesn't use any sliding window to train story not from the start Training took approximately 9 hours (3 hours per epoch) on 40GB A100. ~30GB VRAM was used I use tokenizer from open llama 3b. However I had troubles with it locally(https://github.com/openlm research/open llama/issues/69). I had no troubles on the cloud machine with preninstalled libraries. Demo script is demo.py Validation script is provided: valid.py. use it like python valid.py path/to/TinyStoriesV2 GPT4 valid.txt [optional model id or path] : After training I decided that i…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy