notesum.ai

Published at May 11

Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model

NeurIPS

Released Date: May 11, 2024

Authors: Haogeng Liu¹, Quanzeng You², Xiaotian Han², Yongfei Liu², Huaibo Huang¹, Ran He¹, Hongxia Yang¹

Aff.: ¹MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences; ²ByteDance, Inc

Arxiv: https://openreview.net/pdf/4041aad3b42874372a267d6990a28307a3c622bb.pdf

Refer to caption

Benchmark	Description of the task	Metric
TextVQA singh2019towards	QAs about text in image (Visual Perception)	VQA score ( $\uparrow$ )
VizWiz VQA gurari2018vizwiz	QAs about image from blinds (Visual Perception)	VQA score ( $\uparrow$ )
GQA hudson2019gqa	QAs of real world comprehension and complex reasoning	EM ( $\uparrow$ )
VQAv2 VQA	QAs require vision, language and prior world knowledge	VQA score ( $\uparrow$ )
POPE li2023evaluating	QAs for Object Hallucination evaluation	F1 Score ( $\uparrow$ )
Sci-QA(Img) lu2022learn	QAs about Science	Accuracy ( $\uparrow$ )
MME fu2023mme	Comprehensive Evaluation Benchmark for MLLMs	Accuracy ( $\uparrow$ )
MMbench liu2023mmbench	Comprehensive Evaluation Benchmark for MLLMs	Accuracy ( $\uparrow$ )
MM-Vet yu2023mmvet	Integrated Capabilities Benchmark for MLLMs	GPT-4 score( $\uparrow$ )