PreFLMR M2KR Dataset Card

Dataset details

Dataset type: M2KR is a benchmark dataset for multimodal knowledge retrieval. It contains a collection of tasks and datasets for training and evaluating multimodal knowledge retrieval models.

We pre-process the datasets into a uniform format and write several task-specific prompting instructions for each dataset. The details of the instruction can be found in the paper. The M2KR benchmark contains three types of tasks:

Image to Text (I2T) retrieval

These tasks evaluate the ability of a retriever to find relevant documents associated with an input image.
Component tasks are WIT, IGLUE-en, KVQA, and CC3M.

Question to Text (Q2T) retrieval

This task is based on MSMARCO and is included to assess whether multi-modal retrievers retain their ability in text-only retrieval after any retraining for images.

Image & Question to Text (IQ2T) retrieval

This is the most challenging task which requires joint understanding of questions and images for accurate retrieval. It consists of these subtasks:
OVEN, LLaVA, OKVQA, Infoseek and E-VQA.

Paper or resources for more information:

• Paper: https://arxiv.org/abs/2402.08327
• Project Page: https://preflmr.github.io/
• Huggingface Implementation: https://github.com/LinWeizheDragon/FLMR For details on the example usage of the dataset, please see the M2KR Benchmark Datasets

We release the raw images used in M2KR benchmark, please see the M2kR Benchmark Images

License: MIT License

Where to send questions or comments about the model: https://github.com/LinWeizheDragon/FLMR/issues

Intended use

Primary intended uses: The primary use of M2KR is for pretraining general-purpose multimodal knowledge retrieval models and benchmarking their performance.

Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.

Citation If our work helped your research, please kindly cite our paper for PreFLMR.

       
@inproceedings{lin-etal-2024-preflmr,
    title = "{P}re{FLMR}: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers",
    author = "Lin, Weizhe  and
      Mei, Jingbiao  and
      Chen, Jinghong  and
      Byrne, Bill",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.289",
    pages = "5294--5316",
    abstract = "Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pre-trained version of the recently developed Fine-grained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in general-purpose multi-modal retrievers.",
}