notesum.ai

Published at December 9

The Narrow Gate: Localized Image-Text Communication in Vision-Language Models

cs.CV

cs.LG

Released Date: December 9, 2024

Authors: Alessandro Serra, Francesco Ortu¹, Emanuele Panizon², Lucrezia Valeriani¹, Lorenzo Basile¹, Alessio Ansuini², Diego Doimo², Alberto Cazzaniga²

Aff.: ¹University of Trieste, Trieste, Italy; ²AREA Science Park, Trieste, Italy

Arxiv: http://arxiv.org/pdf/2412.06646v1

Model	Ablation	VQAv2	Flickr	MS-COCO	ImageNet
Chameleon-7B	-	$0.51$	$0.23$	$0.35$	$0.46$
	[EOI]	$\mathbf{0.25}$	$\mathbf{0.04}$	$\mathbf{0.03}$	$\mathbf{0.01}$
	${\text{img}\rightarrow\text{text}}$	$0.40$	$0.13$	$0.21$	$0.47$
	random	$0.51$	$0.21$	$0.31$	$0.46$
Chameleon-34B	-	$0.59$	$0.38$	$0.42$	$0.43$
	[EOI]	$\mathbf{0.39}$	$\mathbf{0.02}$	$\mathbf{0.02}$	$\mathbf{0.04}$
	${\text{img}\rightarrow\text{text}}$	$0.50$	$0.25$	$0.28$	$0.42$
	random	$0.58$	$0.34$	$0.38$	$0.42$
Pixtral	-	$0.79$	$0.51$	$0.60$	$0.63$
	[EOI]	$0.77$	$0.53$	$0.58$	$0.61$
	${\text{img}\rightarrow\text{text}}$	$0.37$	$0.00$	$0.01$	$0.06$
	random	$0.77$	$0.46$	$0.55$	$0.61$