notesum.ai

Published at November 15

Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination

cs.CV

cs.AI

Released Date: November 15, 2024

Authors: Haojie Zheng¹, Tianyang Xu², Hanchi Sun³, Shu Pu⁴, Ruoxi Chen³, Lichao Sun³

Aff.: ¹University of Pennsylvania; ²Columbia University; ³Lehigh University; ⁴Independent Researcher

Arxiv: http://arxiv.org/abs/2411.12591v1

Models	Method	MMVP	HallusionBench	POPE_adversarial	Mathvista	SEED-Bench_single	Average	MME
Models	Method	MMVP	HallusionBench	POPE_adversarial	Mathvista	SEED-Bench_single	Average	Perception	Cognition
Baselines	Human	0.957	0.986	0.995	0.603	0.967	0.901	-	-
Baselines	Random choice	0.250	0.500	0.500	0.179	0.250	0.336	-	-
GPT-4o mini	Origin	0.446	0.574	0.786	0.526	0.636	0.590	1097.23	407.14
	zero-shot CoT	0.443 _{( $\downarrow$ 2.85%)}	0.611 _{( $\uparrow$ 6.46%)}	0.773 _{( $\downarrow$ 1.65%)}	0.520 _{( $\downarrow$ 1.14%)}	0.660 _{( $\uparrow$ 3.77%)}	0.601 _{( $\uparrow$ 1.86%)}	1069.81 _{( $\downarrow$ 2.50%)}	417.14 _{( $\uparrow$ 2.46%)}
	VIC	0.520 _{( $\uparrow$ 16.59%)}	0.639 _{( $\uparrow$ 11.37%)}	0.793 _{( $\uparrow$ 0.89%)}	0.536 _{( $\uparrow$ 1.90%)}	0.696 _{( $\uparrow$ 9.43%)}	0.637 _{( $\uparrow$ 7.96%)}	1105.27 _{( $\uparrow$ 0.73%)}	505.00 _{( $\uparrow$ 24.04%)}
Gemini 1.5 Flash	Origin	0.527	0.560	0.769	0.479	0.658	0.599	1077.36	358.92
	zero-shot CoT	0.513 _{( $\downarrow$ 2.54%)}	0.614 _{( $\uparrow$ 9.69%)}	0.741 _{( $\downarrow$ 3.64%)}	0.520 _{( $\uparrow$ 8.56%)}	0.672 _{( $\uparrow$ 2.13%)}	0.612 _{( $\uparrow$ 2.17%)}	1105.9 _{( $\uparrow$ 2.65%)}	500.71 _{( $\uparrow$ 39.50%)}
	VIC	0.553 _{( $\uparrow$ 5.05%)}	0.638 _{( $\uparrow$ 13.93%)}	0.780 _{( $\uparrow$ 1.43%)}	0.516 _{( $\uparrow$ 7.72%)}	0.713 _{( $\uparrow$ 8.36%)}	0.640 _{( $\uparrow$ 6.84%)}	1118.54 _{( $\uparrow$ 3.82%)}	508.21 _{( $\uparrow$ 41.59%)}
GPT-4o	Origin	0.673	0.626	0.811	0.597	0.657	0.673	1174.39	522.85
	zero-shot CoT	0.687 _{( $\uparrow$ 1.99%)}	0.673 _{( $\uparrow$ 7.49%)}	0.793 _{( $\downarrow$ 2.22%)}	0.622 _{( $\uparrow$ 4.19%)}	0.739 _{( $\uparrow$ 12.48%)}	0.701 _{( $\uparrow$ 4.16%)}	1166.12 _{( $\downarrow$ 0.70%)}	537.14 _{( $\uparrow$ 2.73%)}
	VIC	0.747 _{( $\uparrow$ 10.90%)}	0.692 _{( $\uparrow$ 10.52%)}	0.827 _{( $\uparrow$ 1.27%)}	0.620 _{( $\uparrow$ 3.85%)}	0.751 _{( $\uparrow$ 14.31%)}	0.727 _{( $\uparrow$ 8.08%)}	1238.69 _{( $\uparrow$ 5.48%)}	557.85 _{( $\uparrow$ 6.69%)}
Gemini 1.5 Pro	Origin	0.420	0.617	0.779	0.568	0.678	0.612	1166.82	462.14
	zero-shot CoT	0.500 _{( $\uparrow$ 19.05%)}	0.611 _{( $\downarrow$ 0.97%)}	0.793 _{( $\uparrow$ 1.67%)}	0.563 _{( $\downarrow$ 0.88%)}	0.691 _{( $\uparrow$ 1.92%)}	0.632 _{( $\uparrow$ 3.20%)}	1099.33 _{( $\downarrow$ 5.78%)}	521.42 _{( $\uparrow$ 12.83%)}
	VIC	0.553 _{( $\uparrow$ 31.74%)}	0.664 _{( $\uparrow$ 7.62%)}	0.803 _{( $\uparrow$ 3.08%)}	0.558 _{( $\downarrow$ 1.76%)}	0.713 _{( $\uparrow$ 5.16%)}	0.658 _{( $\uparrow$ 7.55%)}	1147.23 _{( $\downarrow$ 1.68%)}	561.42 _{( $\uparrow$ 21.48%)}