notesum.ai

Published at October 31

Nearest Neighbor Normalization Improves Multimodal Retrieval

cs.CV

cs.AI

cs.CL

Released Date: October 31, 2024

Authors: Neil Chowdhury¹, Franklin Wang¹, Sumedh Shenoy¹, Douwe Kiela², Sarah Schwettmann¹, Tristan Thrush²

Aff.: ¹Massachusetts Institute of Technology; ²Stanford University

Arxiv: http://arxiv.org/abs/2410.24114v1

Refer to caption

	Flickr30k retrieval				COCO retrieval
	Original	DBNorm	NNN Flickr	NNN COCO	Original	DBNorm	NNN Flickr	NNN COCO
CLIP	79.30	81.20 (+1.9)	81.20 (+1.9)	80.10 (+0.8)	50.02	53.20 (+3.2)	51.60 (+1.6)	53.66 (+3.6)
CLIP ft. Flickr	85.70	86.50 (+0.8)	87.30 (+1.6)	86.60 (+0.9)	53.74	55.42 (+1.7)	53.92 (+0.2)	56.44 (+2.7)
CLIP ft. COCO	82.10	81.90 (-0.2)	82.80 (+0.7)	82.70 (+0.6)	63.74	64.72 (+1.0)	63.88 (+0.1)	65.26 (+1.5)
BLIP ft. Flickr	93.40	95.70 (+2.3)	95.20 (+1.8)	94.30 (+0.9)	72.26	78.28 (+6.0)	75.90 (+3.6)	78.30 (+6.0)
BLIP ft. COCO	93.70	94.70 (+1.0)	95.30 (+1.6)	94.60 (+0.9)	79.62	82.52 (+2.9)	79.58 (-0.0)	82.46 (+2.8)
ALBEF ft. Flickr	92.40	93.10 (+0.7)	92.60 (+0.2)	92.70 (+0.3)	69.82	74.62 (+4.8)	71.06 (+1.2)	74.44 (+4.6)
ALBEF ft. COCO	87.30	90.50 (+3.2)	90.00 (+2.7)	89.30 (+2.0)	78.60	80.54 (+1.9)	79.10 (+0.5)	80.68 (+2.1)
SigLIP	89.00	91.60 (+2.6)	91.30 (+2.3)	91.30 (+2.3)	65.32	69.14 (+3.8)	66.80 (+1.5)	69.86 (+4.5)
BEiT-3	89.10	90.70 (+1.6)	91.80 (+2.7)	90.90 (+1.8)	61.12	68.94 (+7.8)	65.66 (+4.5)	69.12 (+8.0)
BEiT-3 ft. Flickr	96.30	94.40 (-1.9)	95.60 (-0.7)	95.90 (-0.4)	72.02	75.12 (+3.1)	72.62 (+0.6)	75.22 (+3.2)
BEiT-3 ft. COCO	93.60	94.50 (+0.9)	95.30 (+1.7)	94.80 (+1.2)	80.72	79.90 (-0.8)	80.42 (-0.3)	81.26 (+0.5)
BEiT-3 Large	91.10	93.20 (+2.1)	93.20 (+2.1)	92.20 (+1.1)	63.26	71.06 (+7.8)	67.60 (+4.3)	71.08 (+7.8)
BEiT-3 Large ft. Flickr	97.20	96.80 (-0.4)	97.20 (0.0)	97.50 (+0.3)	74.32	77.56 (+3.2)	74.86 (+0.5)	77.92 (+3.6)
BEiT-3 Large ft. COCO	95.50	95.00 (0.0)	95.30 (-0.2)	96.20 (+0.7)	82.10	80.88 (-1.2)	81.98 (-0.1)	82.72 (+0.6)