notesum.ai

Published at November 5

Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection

cs.CV

cs.AI

cs.LG

Released Date: November 5, 2024

Authors: Geng Yu¹, Jianing Zhu², Jiangchao Yao³, Bo Han⁴

Aff.: ¹CMIC, Shanghai Jiao Tong University; ²TMLR Group, Hong Kong Baptist University; ³CMIC, Shanghai Jiao Tong University; Shanghai A I Laboratory; ⁴TMLR Group, Hong Kong Baptist University; RIKEN Center for Advanced Intelligence Project

Arxiv: http://arxiv.org/abs/2411.03359v1

Method	OOD dataset
	iNaturalist		SUN		Places365		Textures		Average
	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$
	Zero-shot methods
MCM	31.86	94.17	37.28	92.55	42.94	90.09	58.37	85.83	42.61	90.66
GL-MCM	15.16	96.71	29.16	93.41	37.07	90.37	58.85	83.11	35.06	90.90
	CLIP-based post-hoc methods
MSP^†	74.57	77.74	76.95	73.97	79.72	72.18	73.66	74.84	74.98	76.22
ODIN ^†	98.93	57.73	88.72	78.42	87.80	76.88	85.47	71.49	90.23	71.13
Energy^†	64.98	87.18	46.42	91.17	57.40	87.33	50.39	88.22	54.80	88.48
ReAct^†	65.57	86.87	46.17	91.04	56.85	87.42	49.88	88.13	54.62	88.37
MaxLogit^†	60.88	88.03	44.83	91.16	55.54	87.45	48.72	88.63	52.49	88.82
	Prompt tuning based methods
	1-shot
CoOp	$43.80^{\pm 4.33}$	$91.40^{\pm 0.70}$	$35.42^{\pm 1.56}$	$92.65^{\pm 0.53}$	$40.70^{\pm 2.63}$	$90.49^{\pm 0.68}$	$49.61^{\pm 3.03}$	$87.95^{\pm 0.79}$	$42.38^{\pm 1.92}$	$90.63^{\pm 0.38}$
LoCoOp	$28.81^{\pm 2.78}$	$94.05^{\pm 0.72}$	$25.76^{\pm 0.53}$	$94.51^{\pm 0.19}$	$33.68^{\pm 0.62}$	$91.59^{\pm 0.23}$	$51.53^{\pm 0.42}$	$86.85^{\pm 0.24}$	$34.94^{\pm 0.65}$	$91.75^{\pm 0.23}$
IDLike	$12.07^{\pm 0.88}$	$97.65^{\pm 0.10}$	$40.55^{\pm 5.84}$	$91.07^{\pm 1.80}$	$47.94^{\pm 5.24}$	$88.31^{\pm 2.05}$	$38.34^{\pm 13.39}$	$89.67^{\pm 4.03}$	$34.72^{\pm 0.80}$	$91.67^{\pm 0.07}$
NegPrompt	$65.03^{\pm 8.69}$	$84.56^{\pm 2.52}$	$44.39^{\pm 1.66}$	$89.63^{\pm 0.66}$	$51.31^{\pm 6.21}$	$86.55^{\pm 2.19}$	$87.60^{\pm 1.61}$	$63.76^{\pm 3.02}$	$62.08^{\pm 3.71}$	$81.13^{\pm 1.78}$
LSN	$59.28^{\pm 7.02}$	$87.20^{\pm 3.15}$	$40.15^{\pm 0.82}$	$91.47^{\pm 0.14}$	$46.11^{\pm 1.86}$	$88.74^{\pm 0.57}$	$60.34^{\pm 0.14}$	$83.92^{\pm 0.42}$	$51.47^{\pm 1.53}$	$87.84^{\pm 0.58}$
SCT	$19.16^{\pm 1.15}$	$95.70^{\pm 0.28}$	$23.52^{\pm 1.91}$	$94.58^{\pm 0.56}$	$32.81^{\pm 1.14}$	$91.23^{\pm 0.32}$	$48.87^{\pm 1.38}$	$86.66^{\pm 0.33}$	$\textbf{31.09}^{\pm 0.48}$	$\textbf{92.04}^{\pm 0.25}$
	16-shot
CoOp	$28.25^{\pm 4.68}$	$93.92^{\pm 1.17}$	$31.15^{\pm 0.89}$	$93.13^{\pm 0.38}$	$39.12^{\pm 1.05}$	$90.50^{\pm 0.37}$	$41.86^{\pm 1.88}$	$90.40^{\pm 0.54}$	$35.09^{\pm 1.60}$	$91.99^{\pm 0.42}$
LoCoOp	$17.58^{\pm 2.22}$	$96.30^{\pm 0.57}$	$22.82^{\pm 0.34}$	$95.20^{\pm 0.06}$	$32.21^{\pm 0.53}$	$92.03^{\pm 0.20}$	$45.27^{\pm 0.95}$	$88.86^{\pm 0.26}$	$29.47^{\pm 0.29}$	$93.10^{\pm 0.03}$
IDLike	$9.71^{\pm 0.60}$	$98.05^{\pm 0.07}$	$38.93^{\pm 0.10}$	$90.54^{\pm 0.68}$	$47.06^{\pm 1.44}$	$88.06^{\pm 0.90}$	$32.82^{\pm 5.12}$	$91.89^{\pm 1.49}$	$32.12^{\pm 1.09}$	$92.14^{\pm 0.01}$
NegPrompt	$37.79^{\pm 0.11}$	$90.49^{\pm 0.01}$	$32.11^{\pm 3.77}$	$92.25^{\pm 1.00}$	$35.52^{\pm 0.41}$	$91.16^{\pm 0.03}$	$43.93^{\pm 9.09}$	$88.38^{\pm 3.31}$	$37.34^{\pm 1.41}$	$90.57^{\pm 0.59}$
LSN	$36.17^{\pm 4.81}$	$92.66^{\pm 1.16}$	$34.27^{\pm 0.44}$	$93.53^{\pm 0.20}$	$41.47^{\pm 0.85}$	$90.52^{\pm 0.37}$	$46.43^{\pm 0.60}$	$89.38^{\pm 0.24}$	$39.58^{\pm 0.73}$	$91.53^{\pm 0.09}$
SCT	$13.94^{\pm 0.68}$	$95.86^{\pm 0.28}$	$20.55^{\pm 1.07}$	$95.33^{\pm 0.12}$	$29.86^{\pm 0.67}$	$92.24^{\pm 0.05}$	$41.51^{\pm 0.48}$	$89.06^{\pm 0.09}$	$\textbf{26.47}^{\pm 0.39}$	$\textbf{93.37}^{\pm 0.07}$