notesum.ai

Published at October 23

ROCKET-1: Master Open-World Interaction with Visual-Temporal Context Prompting

cs.CY

cs.AI

cs.HC

cs.IR

Released Date: October 23, 2024

Authors: Shaofei Cai¹, Zihao Wang¹, Kewei Lian¹, Zhancun Mu¹, Xiaojian Ma², Anji Liu³, Yitao Liang⁴

Aff.: ¹PKU; ²BIGAI; ³UCLA; ⁴B

Arxiv: https://arxiv.org/abs/2410.17856v1

Refer to caption

Method	Prompt	Hunt		Mine		Interact		Navigate		Tool		Place		Overall
Method	Prompt													Overall
VPT-bc	N/A	13	16	0	13	3	31	0	9	0	0	0	0	7
STEVE-1	Human	0	6	0	69	0	3	0	31	91	6	16	0	19
GROOT-1	Human	9	22	0	6	3	6	0	3	47	13	3	0	9
\hdashline[1pt/3pt] ROCKET-1	Molmo	$\mathbf{91}_{+82}$	$\mathbf{84}_{+62}$	$\mathbf{78}_{+78}$	$\mathbf{75}_{+6}$	$\mathbf{81}_{+78}$	$\mathbf{50}_{+19}$	$\mathbf{78}_{+78}$	$\mathbf{97}_{+66}$	$\mathbf{94}_{+3}$	$\mathbf{91}_{+78}$	$\mathbf{72}_{+56}$	$\mathbf{91}_{+91}$	$\mathbf{82}_{+63}$
ROCKET-1	Human	$\mathbf{94}_{+85}$	$\mathbf{91}_{+69}$	$\mathbf{91}_{+91}$	$\mathbf{94}_{+25}$	$\mathbf{94}_{+91}$	$\mathbf{91}_{+60}$	$\mathbf{97}_{+97}$	$\mathbf{97}_{+66}$	$\mathbf{97}_{+6}$	$\mathbf{97}_{+84}$	$\mathbf{94}_{+78}$	$\mathbf{97}_{+97}$	$\mathbf{95}_{+76}$