notesum.ai

Published at November 27

Optimizing Multispectral Object Detection: A Bag of Tricks and Comprehensive Benchmarks

cs.CV

Released Date: November 27, 2024

Authors: Chen Zhou¹, Peng Cheng¹, Junfeng Fang¹, Yifan Zhang¹, Yibo Yan¹, Xiaojun Jia¹, Yanyan Xu¹, Kun Wang¹, Xiaochun Cao¹

Aff.: ¹Not provided

Arxiv: http://arxiv.org/abs/2411.18288v1

Refer to caption

Backbone	Method	Datasets		Fusion Strategy
				Pixel-Fusion	Feature-Fusion	Decision-Fusion	RGB-Output	TIR-Output
Resnet50	YOLO-V5 [86]	KAIST		$29.31_{\pm 1.21}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}15.17_{\pm 1.59}}}}$	$17.37_{\pm 2.24}$	$18.39_{\pm 1.75}$	$17.89_{\pm 2.92}$
		FLIR		$63.53_{\pm 2.81}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}73.22_{\pm 1.66}}}}$	$68.71_{\pm 1.20}$	$67.84_{\pm 1.17}$	$68.23_{\pm 1.75}$
	CO-DETR [92]	KAIST		$26.17_{\pm 2.01}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}14.67_{\pm 2.35}}}}$	$16.83_{\pm 1.17}$	$17.65_{\pm 2.55}$	$17.17_{\pm 1.57}$
		FLIR		$62.39_{\pm 2.65}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}78.97_{\pm 2.50}}}}$	$69.36_{\pm 2.40}$	$68.93_{\pm 2.12}$	$68.35_{\pm 2.74}$
	RTMDET [93]	KAIST		$23.59_{\pm 1.64}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}14.13_{\pm 2.58}}}}$	$18.97_{\pm 1.15}$	$17.36_{\pm 1.85}$	$16.33_{\pm 1.45}$
		FLIR		$57.81_{\pm 1.97}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}75.36_{\pm 2.31}}}}$	$66.29_{\pm 1.71}$	$64.32_{\pm 2.44}$	$63.97_{\pm 2.12}$
	DINO [94]	KAIST		$27.73_{\pm 1.98}$	$18.93_{\pm 2.16}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}16.67_{\pm 1.12}}}}$	$19.57_{\pm 2.66}$	$17.98_{\pm 1.81}$
		FLIR		$57.69_{\pm 2.87}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}76.83_{\pm 2.33}}}}$	$69.15_{\pm 2.14}$	$67.96_{\pm 2.02}$	$67.71_{\pm 2.33}$
Vit-L	YOLO-V5 [86]	KAIST		$28.99_{\pm 1.98}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}13.52_{\pm 2.18}}}}$	$17.61_{\pm 2.32}$	$18.02_{\pm 2.45}$	$18.63_{\pm 2.66}$
		FLIR		$62.72_{\pm 1.91}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}72.67_{\pm 2.71}}}}$	$68.72_{\pm 1.99}$	$68.12_{\pm 2.47}$	$68.61_{\pm 2.99}$
	CO-DETR [92]	KAIST		$27.95_{\pm 2.53}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}13.63_{\pm 1.53}}}}$	$16.85_{\pm 2.34}$	$18.36_{\pm 1.90}$	$17.17_{\pm 2.97}$
		FLIR		$63.30_{\pm 2.89}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}76.55_{\pm 2.35}}}}$	$70.72_{\pm 1.52}$	$67.27_{\pm 2.95}$	$69.65_{\pm 2.11}$
	RTMDET [93]	KAIST		$22.60_{\pm 2.32}$	$15.11_{\pm 2.65}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}14.53_{\pm 2.95}}}}$	$16.34_{\pm 2.15}$	$16.19_{\pm 2.98}$
		FLIR		$56.75_{\pm 2.78}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}74.39_{\pm 2.19}}}}$	$66.79_{\pm 1.61}$	$65.63_{\pm 2.22}$	$65.07_{\pm 1.68}$
	DINO [94]	KAIST		$26.97_{\pm 2.68}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}12.21_{\pm 2.95}}}}$	$15.54_{\pm 1.95}$	$19.89_{\pm 1.85}$	$18.08_{\pm 2.34}$
		FLIR		$56.11_{\pm 1.89}$	$\mathbf{\underline{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}% {1,0,0}77.12_{\pm 1.99}}}}$	$70.61_{\pm 1.62}$	$68.37_{\pm 2.95}$	$69.97_{\pm 2.11}$