gLM2 650M gLM2 is a mixed modality genomic language model, trained on the OMG Dataset . The model encodes a genomic scaffold with both both amino acid and DNA tokens. gLM2 is trained at two scales: 150M (available at tattabio/gLM2 150M ) and 650M parameters. See https://github.com/TattaBio/gLM2 for inference scripts. Model Description gLM2 is a transformer encoder trained with the masked language modeling objective. It encodes a genomic contig as a sequence of protein coding sequences (CDS) and DNA inter genic sequences (IGS). CDS elements are tokenized using per amino acid tokens, and IGS elements are tokenized using per nucleotide tokens. To encode the genomic strand, we prepended each genomic element with a special token, either or to indicate the positive and negative strands. To avoid collision between amino acid and nucleotide tokens, the tokenizer expects all amino acids to be uppercase, and all nucleotides to be lowercase. UPDATE(09/2024): We updated the model with longer context length (4096 tokens vs. 2048 tokens) and per nucleotide IGS tokenization instead of BPE. Getting Started Training Data gLM2 is trained on the OMG dataset. To improve the dataset balance and remove…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy