ByT5 Base ByT5 is a tokenizer free version of Google's T5 and generally follows the architecture of MT5. ByT5 was only pre trained on mC4 excluding any supervised training with an average span mask of 20 UTF 8 characters. Therefore, this model has to be fine tuned before it is useable on a downstream task. ByT5 works especially well on noisy text data, e.g. , google/byt5 base significantly outperforms mt5 base on TweetQA. Paper: ByT5: Towards a token free future with pre trained byte to byte models Authors: Linting Xue, Aditya Barua, Noah Constant, Rami Al Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel Example Inference ByT5 works on raw UTF 8 bytes and can be used without a tokenizer: For batched inference & training it is however recommended using a tokenizer class for padding: Abstract Most widely used pre trained language models operate on sequences of tokens corresponding to word or subword units. Encoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from the model. Token free models that instead operate directly on raw text (bytes or characters) have many benefits: they can process text in any language o…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy