notesum.ai

Published at October 21

How to Build a Pre-trained Multimodal model for Simultaneously Chatting and Decision-making?

cs.AI

Released Date: October 21, 2024

Authors: Zuojin Tang¹, Bin Hu², Chenyang Zhao², De Ma¹, Gang Pan¹, Bin Liu³

Aff.: ¹College of Computer Science and Technology, Zhejiang University; ²Zhejiang Lab; ³CRC, Midea Group

Arxiv: https://arxiv.org/abs/2410.15885v1

Refer to caption

Method	Input	DS $\uparrow$	AR ( $f$ ) $\uparrow$	ASD(m) $\uparrow$	ER(%) $\uparrow$	OR(%) $\downarrow$	CR(%) $\downarrow$
BC	image	$36.39\pm 13.37$	$314.66\pm 86.02$	$64.08\pm 10.48$	$9.04\pm 0.62$	$37.56\pm 16.44$	$45.00\pm 11.41$
Dreamer	image	$-0.03\pm 0.01$	$-14.96\pm 0.09$	$0.02\pm 0.01$	$0.22\pm 0.01$	$0.00\pm 0.00$	$0.00\pm 0.00$
Forbes	image	$0.98\pm 1.43$	$21.63\pm 21.72$	$22.84\pm 1.00$	$6.30\pm 0.31$	$18.78\pm 1.03$	$56.67\pm 9.20$
DT	image	$7.68\pm 3.24$	$51.97\pm 29.33$	$23.74\pm 2.47$	$9.92\pm 0.71$	$10.31\pm 2.32$	$65.00\pm 10.94$
DriverGPT4	image, text	-	-	-	-	-	-
Openvla	image, text	$-7.84\pm 0.67$	$-160.37\pm 7.85$	$18.03\pm 1.92$	$4.76\pm 0.19$	$20.77\pm 3.36$	$100.00\pm 0.00$
VLA4CD	image, text	$105.25\pm 14.03$	$349.52\pm 49.75$	$59.76\pm 5.04$	$25.02\pm 2.57$	$19.93\pm 2.11$	$30.00\pm 10.51$