notesum.ai

Published at November 27

XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration

cs.CV

cs.LG

Released Date: November 27, 2024

Authors: Denys Rozumnyi¹, Nadine Bertsch¹, Othman Sbai¹, Filippo Arcadu¹, Yuhua Chen¹, Artsiom Sanakoyeu¹, Manoj Kumar¹, Catherine Herold¹, Robin Kips¹

Aff.: ¹Meta Reality Labs Zurich

Arxiv: http://arxiv.org/abs/2411.18377v1

Refer to caption

Model	MPJPE (cm) up $\|$ low	MPJRE (radians) up $\|$ low	MPJVE ( $m^{2}$ / $s^{3}$ ) up $\|$ low	jitter (pred/gt) up $\|$ low	PC-loss
All Motions
AGRoL [11] (retrained)	1.87 $\|$ 11.21	1.79 $\|$ 2.30	10.27 $\|$ 34.36	2.55 $\|$ 6.19	0.61
+ MPE	1.78 $\|$ 9.62	1.75 $\|$ 2.23	9.94 $\|$ 31.89	3.22 $\|$ 12.66	0.39
+ MPE + SPC-decoder	1.77 $\|$ 9.30	1.75 $\|$ 2.21	9.95 $\|$ 32.41	3.11 $\|$ 12.98	0.34
+ MPE + SPC-decoder + PC-loss	1.78 $\|$ 10.25	1.76 $\|$ 2.25	9.96 $\|$ 32.97	2.84 $\|$ 9.43	0.48
+ MPE + SPC-decoder + SPC-loss	1.76 $\|$ 9.27	1.74 $\|$ 2.21	9.78 $\|$ 31.40	2.88 $\|$ 9.47	0.32
Kicking
AGRoL [11] (retrained)	1.58 $\|$ 10.21	1.63 $\|$ 2.05	10.53 $\|$ 41.10	1.87 $\|$ 3.32	0.80
+ MPE	1.49 $\|$ 8.27	1.61 $\|$ 2.01	10.22 $\|$ 36.56	2.31 $\|$ 7.6	0.39
+ MPE + SPC-decoder	1.48 $\|$ 7.97	1.61 $\|$ 1.99	10.24 $\|$ 37.76	2.23 $\|$ 7.88	0.33
+ MPE + SPC-decoder + PC-loss	1.50 $\|$ 8.91	1.62 $\|$ 2.02	10.26 $\|$ 38.98	2.08 $\|$ 5.70	0.51
+ MPE + SPC-decoder + SPC-loss	1.48 $\|$ 7.78	1.59 $\|$ 1.98	10.11 $\|$ 36.42	2.18 $\|$ 5.93	0.28