Perceiver IO for language Perceiver IO model pre trained on the Masked Language Modeling (MLM) task proposed in BERT using a large text corpus obtained by combining English Wikipedia and C4. It was introduced in the paper Perceiver IO: A General Architecture for Structured Inputs & Outputs by Jaegle et al. and first released in this repository. Disclaimer: The team releasing Perceiver IO did not write a model card for this model so this model card has been written by the Hugging Face team. Model description Perceiver IO is a transformer encoder model that can be applied on any modality (text, images, audio, video, ...). The core idea is to employ the self attention mechanism on a not too large set of latent vectors (e.g. 256 or 512), and only use the inputs to perform cross attention with the latents. This allows for the time and memory requirements of the self attention mechanism to not depend on the size of the inputs. To decode, the authors employ so called decoder queries, which allow to flexibly decode the final hidden states of the latents to produce outputs of arbitrary size and semantics. For masked language modeling, the output is a tensor containing the prediction scores…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy