Using Caduceus To use the pre trained model for masked language modeling, use the following snippet: Alternatively, you can instantiate a model from scratch to train on your own data as follows: Model Details This is the Caduceus PS model with hidden dimension 256 and 16 MambaDNA layers. This model is reverse complement (RC) equivariant and thus no RC data augmentation is required when training this model, either during pre training or for downstream fine tuning. Note that the model hidden state will be twice that of a non RC equivariant counterpart. For downstream task training and inference, and to ensure RC invariant outputs at downstream time, one can either run the downstream model on the hidden state and its RC or one can take the hidden state and its RC and average them before passing to the downstream model. To RC the hidden states, one can use: hidden states.flip(dim=( 2, 1)) which will flip along the sequence lenght and channel dimensions. This model was pre trained on the human reference genome with sequence length 131,072 for 50k steps (each step contained ~1M base pairs / tokens). For more details, please see our paper: Caduceus: Bi Directional Equivariant Long Range D…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy