ColPali: Visual Retriever based on PaliGemma 3B with ColBERT strategy ColPali is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features. It is a PaliGemma 3B extension that generates ColBERT style multi vector representations of text and images. It was introduced in the paper ColPali: Efficient Document Retrieval with Vision Language Models and first released in this repository Model Description This model is built iteratively starting from an off the shelf SigLIP model. We finetuned it to create BiSigLIP and fed the patch embeddings output by SigLIP to an LLM, PaliGemma 3B to create BiPali. One benefit of inputting image patch embeddings through a language model is that they are natively mapped to a latent space similar to textual input (query). This enables leveraging the ColBERT strategy to compute interactions between text tokens and image patches, which enables a step change improvement in performance compared to BiPali. Model Training Dataset Our training dataset of 127,460 query page pairs is comprised of train sets of openly available academic datasets (63%) and a syn…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy