notesum.ai

Published at November 25

CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction

cs.CV

Released Date: November 25, 2024

Authors: Yuan Zhou¹, Qingshan Xu¹, Jiequan Cui¹, Junbao Zhou¹, Jing Zhang², Richang Hong³, Hanwang Zhang¹

Aff.: ¹Nanyang Technological University; ²Beihang University; ³Hefei University of Technology

Arxiv: http://arxiv.org/abs/2411.16170v1

						OB $\uparrow$			IS $\uparrow$			SS $\uparrow$
Backbone	Ref.	Type	GMACs $\downarrow$	iPhone13 ( $ms$ ) $\downarrow$	RTX 4090 ( $s$ ) $\downarrow$	$\bm{\mathrm{AP}}^{b}$	$\bm{\mathrm{AP}}^{b}_{50}$	$\bm{\mathrm{AP}}^{b}_{75}$	$\bm{\mathrm{AP}}^{m}$	$\bm{\mathrm{AP}}^{m}_{50}$	$\bm{\mathrm{AP}}^{m}_{75}$	mIoU
EfficientFormer-L1 [26]	NeurIPS’ 22	CONV	$6.9$	$4.8$	$0.09$	$37.9$	$60.3$	$41.0$	$35.4$	$57.3$	$37.3$	$38.9$
MobileViG-M [32]	CVPR’ 23	GNN+CONV	$7.8$	$5.0$	$0.12$	$41.3$	$62.8$	$45.1$	$38.1$	$60.1$	$40.8$	n/a
MobileViGv2-M [1]	CVPR’ 24	GNN+CONV	$8.5$	$5.5$	$0.14$	$42.5$	$63.9$	$46.3$	$38.8$	$60.8$	$41.7$	$42.9$
ResNet18 [20]	CVPR’ 16	CONV	$9.5$	$3.2$	$0.04$	$34.0$	$54.0$	$36.7$	$31.2$	$51.0$	$32.7$	$32.9$
FastViT-SA12 [42]	CVPR’ 23	SA+CONV	$8.0$	$6.9$	$0.14$	$38.9$	$60.5$	$42.2$	$35.9$	$57.6$	$38.1$	$38.0$
PoolFormer-S12 [55]	CVPR’ 22	CONV	$9.5$	$5.7$	$0.09$	$37.3$	$59.0$	$40.1$	$34.6$	$55.8$	$36.9$	$37.2$
SLAB-PVT-T [14]	ICLR’ 24	LA	$10.4$	$14.2$	$0.13$	$36.5$	$59.0$	$39.2$	$34.4$	$55.7$	$36.5$	n/a
FLatten-PVT-T [15]	CVPR’ 23	LA	$10.4$	$18.6$	$0.14$	$38.2$	$61.6$	$41.9$	$37.0$	$57.6$	$39.0$	$37.2$
Agent-PVT-T [17]	ECCV’ 24	SA	$10.7$	$17.6$	$0.12$	$41.4$	$64.1$	$45.2$	$38.7$	$61.3$	$41.6$	$40.2$
PoolFormer-S24 [55]	CVPR’ 22	CONV	$17.7$	$9.8$	$0.17$	$40.1$	$62.2$	$43.4$	$37.0$	$59.1$	$39.6$	$40.3$
EfficientFormer-L3 [26]	NeurIPS’ 22	CONV	$21.0$	$10.3$	$0.20$	$41.4$	$63.9$	$44.7$	$38.1$	$61.0$	$40.4$	$43.5$
ResNet50 [20]	CVPR’ 16	CONV	$21.4$	$7.6$	$0.14$	$38.0$	$58.6$	$41.4$	$34.4$	$55.1$	$36.7$	$36.7$
MLLA-T [16]	NeurIPS’ 24	LA+CONV	$22.0$	$25.5$	$0.29$	$46.8$	$69.5$	$51.5$	$42.1$	$66.4$	$45.0$	n/a
Swin-T [28]	CVPR’ 22	SA	$24.2$	$28.2$	$0.28$	$42.2$	$64.4$	$46.2$	$39.1$	$64.6$	$42.0$	$41.5$
FLatten-Swin-T [15]	CVPR’ 23	LA	$24.2$	$28.6$	$0.28$	$44.2$	$67.3$	$48.5$	$40.2$	$63.8$	$43.0$	$44.8$
Agent-Swin-T [17]	ECCV’ 24	SA	$24.2$	$26.8$	$0.26$	$44.6$	$67.5$	$48.7$	$40.7$	$64.4$	$43.4$	$46.7$
SLAB-Swin-T [14]	ICLR’ 24	LA	$24.2$	$26.4$	$0.26$	$43.9$	$66.5$	$48.1$	$40.1$	$63.3$	$43.1$	n/a
ResNet101 [20]	CVPR’ 16	CONV	$40.8$	$11.5$	$0.22$	$40.4$	$61.1$	$44.2$	$36.4$	$57.7$	$38.8$	$38.8$
\hdashline Our CARE-S0	-	LA+CONV	$3.8$	$4.1$	$0.07$	$40.3$	$62.2$	$43.8$	$36.8$	$58.8$	$39.2$	$38.5$
Our CARE-S1	-	LA+CONV	$5.4$	$5.5$	$0.11$	$41.5$	$63.4$	$44.8$	$37.8$	$60.2$	$39.9$	$41.0$
Our CARE-S2	-	LA+CONV	$10.1$	$7.3$	$0.14$	$43.1$	$64.4$	$47.0$	$39.0$	$61.6$	$41.9$	$43.5$