notesum.ai

Published at November 12

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

cs.SD

cs.AI

cs.CV

cs.MM

eess.AS

Released Date: November 12, 2024

Authors: Xinyuan Qian, Jiaran Gao, Yaodan Zhang, Qiquan Zhang, Hexin Liu, Leibny Paola Garcia, Haizhou Li

Arxiv: http://arxiv.org/abs/2411.07751v1

Refer to caption

Dataset	Model	PESQ						STOI (in %)
Dataset	Model	-5 dB	0 dB	5 dB	10 dB	15 dB	Avg.	-5 dB	0 dB	5 dB	10 dB	15 dB	Avg.
MUSIC	Unprocessed	1.15	1.22	1.39	1.64	2.01	1.47	71.08	78.87	85.62	90.34	93.95	83.65
	MP-SENet	1.77	2.15	2.59	2.90	3.29	2.54	85.02	89.61	92.88	94.68	96.07	91.65
	AViTAR	1.95	2.36	2.78	3.09	3.46	2.73	86.96	90.94	93.94	95.51	96.70	92.81
	ExtBiMamba	1.97	2.39	2.82	3.13	3.54	2.77	87.56	91.34	94.09	95.65	96.82	93.09
	VC-S²E(our)	2.23	2.66	3.08	3.38	3.73	3.02	89.55	92.61	94.96	96.24	97.18	94.11
AVSpeech	Unprocessed	1.10	1.18	1.29	1.53	1.98	1.44	63.21	73.37	82.42	89.01	93.54	81.11
	MP-SENet	1.44	1.76	2.06	2.57	2.97	2.16	80.74	86.25	88.97	91.36	93.42	88.15
	AViTAR	1.63	1.99	2.30	2.82	3.22	2.39	83.16	88.05	90.36	92.96	94.71	89.85
	ExtBiMamba	1.67	2.05	2.38	2.88	3.24	2.44	84.64	88.78	91.01	92.77	94.15	90.27
	VC-S²E(our)	1.96	2.39	2.72	3.20	3.53	2.76	88.13	90.86	92.90	95.24	96.38	92.70
AudioSet	Unprocessed	1.10	1.20	1.33	1.52	1.88	1.41	62.28	74.14	85.23	88.63	93.95	80.99
	MP-SENet	1.33	1.75	2.19	2.49	2.92	2.14	75.85	86.76	92.33	93.94	96.07	88.99
	AViTAR	1.42	1.89	2.43	2.71	3.14	2.32	78.30	87.89	93.46	94.65	97.14	90.29
	ExtBiMamba	1.60	2.06	2.64	2.92	3.30	2.50	81.43	89.57	94.34	95.48	97.48	91.66
	VC-S²E(our)	1.73	2.10	2.73	3.00	3.39	2.59	83.08	90.03	94.78	95.77	97.61	92.25