Model card for RWKV 4 169M parameters trained on Pile dataset RWKV is a project led by Bo Peng. Learn more about the model architecture in the blogposts from Johan Wind here and here. Learn more about the project by joining the RWKV discord server. Table of contents 0. TL;DR 1. Model Details 2. Usage 3. Citation TL;DR Below is the description from the original repository RWKV is an RNN with transformer level LLM performance. It can be directly trained like a GPT (parallelizable). It's combining the best of RNN and transformer great performance, fast inference, saves VRAM, fast training, "infinite" ctx len, and free sentence embedding. Model Details The details of the architecture can be found on the blogpost mentioned above and the Hugging Face blogpost of the integration. Usage Convert the raw weights to the HF format You can use the convert rwkv checkpoint to hf.py script by specifying the repo id of the original weights, the filename and the output directory. You can also optionally directly push the converted model on the Hub by passing push to hub flag and model name argument to specify where to push the converted weights. Generate text You can use the AutoModelForCausalLM and…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy