Monarch Mixer BERT An 80M checkpoint of M2 BERT, pretrained with sequence length 8192, and it has been fine tuned for long context retrieval. Check out the paper Monarch Mixer: A Simple Sub Quadratic GEMM Based Architecture and our blog post on retrieval for more on how we trained this model for long sequence. This model was trained by Jon Saad Falcon, Dan Fu, and Simran Arora. Check out our GitHub for instructions on how to download and fine tune it! How to use You can load this model using Hugging Face AutoModel : You should expect to see a large error message about unused parameters for FlashFFTConv. If you'd like to load the model with FlashFFTConv, you can check out our GitHub. This model generates embeddings for retrieval. The embeddings have a dimensionality of 768: You can also get embeddings from this model using the Together API as follows (you can find your API key here): Acknowledgments Alycia Lee helped with AutoModel support. Citation If you use this model, or otherwise found our work valuable, you can cite us as follows:
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy