notesum.ai

Published at November 21

Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

cs.CV

Released Date: November 21, 2024

Authors: Honglin Li¹, Yuting Gao², Chenglu Zhu³, Jingdong Chen, Ming Yang², Lin Yang³

Aff.: ¹Zhejiang University; ²Ant Group; ³Westlake University

Arxiv: http://arxiv.org/abs/2411.13909v1

[Uncaptioned image]

Method	LLM	Image	Sample Size		VQA ${}^{\textnormal{v2}}$	GQA	SQA ${}^{\textnormal{I}}$	VQA ${}^{\textnormal{T}}$	POPE	MME	MMB
Method	LLM	Size	Pretrain	Finetune	VQA ${}^{\textnormal{v2}}$	GQA	SQA ${}^{\textnormal{I}}$	VQA ${}^{\textnormal{T}}$	POPE	MME	MMB
BLIP-2	Vicuna-13B	224²	129M	-	65.0	41.0	61.0	42.5	85.3	1293.8	–
InstructBLIP	Vicuna-7B	224²	129M	1.2M	–	34.5	60.5	50.1	–	–	36.0
InstructBLIP	Vicuna-13B	224²	129M	1.2M	–	33.4	63.1	50.7	78.9	1212.8	–
Qwen-VL	Qwen-7B	448²	1.4B	50M	78.8	59.3	67.1	63.8	–	–	38.2
Qwen-VL-Chat	Qwen-7B	448²	1.4B	50M	78.2	57.5	68.2	61.5	–	1487.5	60.6
mPLUG-Owl2	Llama-7B	448²	400M	1.2M	79.4	56.1	-	58.2	85.8	1450.2	64.5
VILA	Llama 2-7B	336²	50M	1M	79.9	62.3	68.2	64.4	85.5	1533.0	68.9
LWM	Llama-2-7B	256²	-	-	55.8	44.8	-	18.8	75.2	-	-
Show-o	Phi-1.5-1.3B	256²	-	-	59.3	48.7	-	-	73.8	948.4	-
VILA-U	Llama-2-7B	384²	-	7M	79.4	60.8	-	60.8	85.8	1401.8	-
LLaVA-TokenPack	Vicuna-7B	336²	558K	665K	77.9	61.9	-	57.2	87.0	-	65.1
Cambrian-dev	Vicuna-7B	^†518²	1.2M	737K	-	63.3	68.8	60.4	-	1432.0	61.3
LLaVA-VisPrompt	Vicuna-7B	336²	558K	665K	79.8	63.3	69.5	59.8	88.9	1515.3	67.6
LLaVA-1.5	Vicuna-7B	336²	558K	665K	78.5	62.0	66.8	58.2	85.9	1510.7	64.3
Panther	Vicuna-7B	336²	558K	665K	80.8	65.2	67.8	59.6	86.7	1507.3	67.1
LLaVA-1.5	Llama 3-8B	336²	558K	665K	79.7	63.3	73.3	58.4	84.6	1506.5	68.9
Panther	Llama 3-8B	336²	558K	665K	81.2	65.5	73.5	59.5	85.7	1513.6	71.9