notesum.ai

Published at November 25

cs.CV

cs.AI

cs.CL

cs.RO

Released Date: November 25, 2024

Authors: Chan Hee Song¹, Valts Blukis², Jonathan Tremblay², Stephen Tyree², Yu Su¹, Stan Birchfield²

Aff.: ¹The Ohio State University; ²NVIDIA

Open-source VLMs
Model	Indoor			Tabletop			Average
Model	Configuration	Context	Compatibility	Configuration	Context	Compatibility	Indoor	Tabletop	Total
2D VLMs
VILA [30]	54.7	18.3	56.3	45.1	13.2	53.8	43.1	37.4	40.2
+RoboSpatial	71.4 $\uparrow$	45.9 $\uparrow$	77.2 $\uparrow$	71.8 $\uparrow$	43.7 $\uparrow$	73.3 $\uparrow$	64.8 $\uparrow$	62.9 $\uparrow$	63.9 $\uparrow$
LLaVA-Next [34]	48.9	12.5	32.7	48.3	8.4	30.9	31.4	29.2	30.3
+RoboSpatial	69.3 $\uparrow$	41.3 $\uparrow$	70.5 $\uparrow$	70.7 $\uparrow$	44.8 $\uparrow$	66.1 $\uparrow$	60.4 $\uparrow$	60.5 $\uparrow$	60.5 $\uparrow$
SpaceLLaVA [8]	52.6	15.3	49.0	66.5	12.2	60.1	38.9	46.2	43.6
+RoboSpatial	76.0 $\uparrow$	50.7 $\uparrow$	76.6 $\uparrow$	74.9 $\uparrow$	46.4 $\uparrow$	70.5 $\uparrow$	67.8 $\uparrow$	63.6 $\uparrow$	65.7 $\uparrow$
RoboPoint [57]	39.0	41.4	38.3	37.9	31.6	45.2	39.6	38.2	38.9
+RoboSpatial	72.2 $\uparrow$	68.9 $\uparrow$	72.1 $\uparrow$	70.3 $\uparrow$	61.7 $\uparrow$	78.4 $\uparrow$	71.0 $\uparrow$	70.1 $\uparrow$	70.6 $\uparrow$
3D VLMs
3D-LLM [18]	54.5	8.1	53.6	59.2	10.6	57.4	37.6	42.4	40.0
+RoboSpatial	76.3 $\uparrow$	35.4 $\uparrow$	77.5 $\uparrow$	76.2 $\uparrow$	46.8 $\uparrow$	75.0 $\uparrow$	63.1 $\uparrow$	66.0 $\uparrow$	64.6 $\uparrow$
LEO [20]	56.1	11.3	58.3	60.8	11.1	59.3	41.9	43.7	42.8
+RoboSpatial	80.2 $\uparrow$	56.7 $\uparrow$	82.5 $\uparrow$	78.1 $\uparrow$	55.2 $\uparrow$	78.9 $\uparrow$	73.1 $\uparrow$	70.7 $\uparrow$	71.9 $\uparrow$
Not available for fine-tuning
2D VLMs
Molmo [12]	40.6	48.2	60.0	61.5	35.8	54.6	49.6	50.6	50.1
GPT-4o [40]	63.5	25.1	59.4	62.3	27.9	66.8	49.3	52.3	50.8