notesum.ai

Published at November 26

cs.MM

cs.CV

cs.SD

eess.AS

Released Date: November 26, 2024

Authors: Akshita Gupta¹, Tatiana Likhomanenko², Karren Dai Yang, Richard He Bai, Zakaria Aldeneh², Navdeep Jaitly²

Aff.: ¹University of Guelph; ²Apple

Method	Input Modality	GT WER ( $\downarrow$ )	GT (discrete) WER ( $\downarrow$ )	WER ( $\downarrow$ )	Sync Score ( $\uparrow$ )	TimeSync (s) ( $\downarrow$ )
TTS	Text	4.0 ±0.1	10.5 ±0.1	19.0 $\mathord{+}8.5$	-	-
VTTS (VT-ordered)	Video-Text	4.0 ±0.1	10.5 ±0.1	17.2 $\mathord{+}6.7$	-	-
TTS	Text	2.6 ±0.1	10.1 ±0.2	14.7 $\mathord{+}4.7$	1.54	0.62 ±0.98
VTTS (TV-streaming)	Text-Video			14.5 $\mathord{+}4.4$	1.66	0.49 ±0.63
VTTS (TV-ordered)	Text-Video			14.1 $\mathord{+}4.4$	1.67	0.44 ±0.65
VTTS (VT-ordered)	Video-Text			12.2 $\mathord{+}2.1$	1.64	0.47 ±0.63