notesum.ai

Published at November 6

RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models

cs.CV

cs.AI

Released Date: November 6, 2024

Authors: Maya Varma¹, Jean-Benoit Delbrouck², Zhihong Chen¹, Akshay Chaudhari¹, Curtis Langlotz¹

Aff.: ¹Stanford University; ²Stanford University; Hugging Face

Arxiv: http://arxiv.org/abs/2411.04097v1

Refer to caption

Method	Stage 1 Discovery Precision@10 $>0.6$				Stage 1 Discovery Precision@10 $>0.8$
	Img. Overall	Img. WG	Reg. Overall	Reg. WG	Img. Overall	Img. WG	Reg. Overall	Reg. WG
Standard FT	64.0	31.4	72.0	46.9	64.6	31.0	72.9	47.4
Upsampled FT	66.6	37.8	74.3	52.2	66.7	37.7	74.7	52.8
VL-ERM	68.8	32.2	75.6	50.3	68.7	30.9	75.9	50.6
VL-GDRO	69.1	33.7	75.6	50.4	68.8	31.1	76.0	51.0
Spurious-Aware	69.8	33.6	76.5	50.6	69.2	30.7	76.8	50.5
RaVL (Ours)	69.8	39.1	78.9	57.8	70.2	40.8	79.5	58.5