notesum.ai

Published at November 14

Cross-Modal Consistency in Multimodal Large Language Models

cs.CL

cs.AI

Released Date: November 14, 2024

Authors: Xiang Zhang¹, Senyu Li², Ning Shi², Bradley Hauer², Zijun Wu², Grzegorz Kondrak², Muhammad Abdul-Mageed³, Laks V. S. Lakshmanan

Aff.: ¹University of Alberta; ²Alberta Machine Intelligence Institute, Dept. of Computing Science, University of Alberta; ³University of British Columbia

Arxiv: http://arxiv.org/abs/2411.09273v1

Task	Modal	Acc	Consistency
MES(Easy)	Text	0.44	0.72
\cdashline2-3	Image	0.24 $\color[rgb]{1,0,0}\Downarrow$	0.72
MES (Hard)	Text	0.62	0.62
\cdashline2-3	Image	0.28 $\color[rgb]{1,0,0}\Downarrow$	0.62
LogicQA	Text	0.64	0.64
\cdashline2-3	Image	0.44 $\color[rgb]{1,0,0}\Downarrow$	0.64
MMLU	Text	1.00	0.74
\cdashline2-3	Image	0.74 $\color[rgb]{1,0,0}\Downarrow$	0.74
TU	Text	0.93	0.10
\cdashline2-3	Image	0.03 $\color[rgb]{1,0,0}\Downarrow$	0.10
MR	Text	0.40	0.92
\cdashline2-3	Image	0.36	0.92
State Machine	Text	0.34	0.67
\cdashline2-3	Image	0.28	0.67