notesum.ai

Published at October 31

An Information Criterion for Controlled Disentanglement of Multimodal Data

cs.LG

cs.AI

cs.IT

math.IT

Released Date: October 31, 2024

Authors: Chenyu Wang¹, Sharut Gupta¹, Xinyi Zhang¹, Sana Tonekaboni², Stefanie Jegelka³, Tommi Jaakkola⁴, Caroline Uhler¹

Aff.: ¹MIT, Broad Institute of MIT and Harvard; ²Broad Institute of MIT and Harvard; ³MIT, TU Munich; ⁴MIT

Arxiv: http://arxiv.org/abs/2410.23996v1

Refer to caption

Dataset	MIMIC	MOSEI	MOSI	UR-FUNNY	MUSTARD
CLIP	64.97 (0.60)	76.87 (0.45)	64.24 (0.88)	62.73 (0.92)	56.04 (4.19)
FactorCL-emb	65.25 (0.45)	71.80 (0.64)	62.97 (0.81)	63.29 (2.07)	56.76 (4.66)
FactorCL-proj	59.43 (1.70)	74.61 (1.65)	56.02 (1.26)	61.25 (0.47)	55.80 (2.18)
FOCAL	64.42 (0.34)	76.77 (0.51)	63.65 (1.09)	62.98 (1.52)	54.35 (0.00)
JointOpt	66.11 (0.64)	76.71 (0.14)	64.24 (1.75)	63.58 (1.45)	56.52 (2.61)
DisentangledSSL (shared)	63.16 (0.48)	76.94 (0.22)	65.16 (0.81)	64.14 (1.53)	54.11 (1.51)
DisentangledSSL (specific)	65.73 (0.09)	75.99 (0.60)	51.70 (0.72)	60.27 (1.28)	61.60 (2.61)
DisentangledSSL (both)	66.44 (0.31)	77.45 (0.06)	65.11 (0.80)	64.24 (1.54)	56.52 (2.18)