notesum.ai

Published at November 8

NeKo: Toward Post Recognition Generative Correction Large Language Models with Task-Oriented Experts

cs.CL

cs.AI

cs.LG

cs.MA

eess.AS

Released Date: November 8, 2024

Authors: Yen-Ting Lin¹, Chao-Han Huck Yang¹, Zhehuai Chen¹, Piotr Zelasko¹, Xuesong Yang¹, Zih-Ching Chen¹, Krishna C Puvvada¹, Szu-Wei Fu¹, Ke Hu¹, Jun Wei Chiu¹, Jagadeesh Balam¹, Boris Ginsburg¹, Yu-Chiang Frank Wang¹

Aff.: ¹NVIDIA

Arxiv: http://arxiv.org/abs/2411.05945v1

Model	AMI	Earnings22	Gigaspeech	LS Clean	LS Other	SPGI	Tedlium	Voxp.	MCV9	Average
End-to-end ASR Methods
Distil-Whisper-V2-Large (Gandhi et al., 2023)	14.65	12.12	10.31	2.95	6.39	3.28	4.30	8.22	12.60	8.31
Whisper-V2-Large (Radford et al., 2022)	16.82	12.02	10.57	2.56	5.16	3.77	4.01	7.50	10.11	8.06
Whisper-V3-Large	16.01	11.30	10.02	2.03	3.91	2.95	3.90	9.52	9.67	7.70
Canary (NVIDIA, 2024)	14.00	12.25	10.19	1.49	2.49	2.06	3.58	5.81	7.75	6.67
Bestow Speech LM (Chen et al., 2024c)	12.58	12.86	10.06	1.64	3.07	2.11	3.41	5.84	6.97	6.50
Cascaded ASR+GEC Methods
Gemma 2B (Team et al., 2024) FFT	13.20	12.30	10.40	1.60	2.60	2.20	3.70	6.00	7.50	6.61
Gemma 8x2B FFT	13.10	12.20	10.30	1.50	2.50	2.10	3.60	5.90	7.40	6.51
NeKo (Ours) Gemma 8x2B	13.00	12.10	10.20	1.40	2.40	2.00	3.50	5.80	7.30	6.41
Mistral 7B (Jiang et al., 2023) FFT	13.07	11.87	10.09	1.48	2.46	2.04	3.55	5.75	7.29	6.40
Mixtral 8x7B (Jiang et al., 2024b) FFT	12.91	12.19	10.34	1.54	2.55	2.12	3.64	5.89	7.43	6.51
Mixtral 8x7B Lora	12.96	12.24	10.38	1.55	2.56	2.13	3.66	5.92	7.47	6.60
Mistral 8x7B Branch-Train-Mix (Sukhbaatar et al., 2024)	13.13	11.93	10.14	1.49	2.47	2.05	3.57	5.78	7.33	6.43
NeKo (Ours) Mixtral 8x7B	12.55	11.82	10.02	1.49	2.47	2.05	3.52	5.76	7.25	6.34
NeKo (Ours) Mixtral 8x22B	12.61	11.93	10.15	1.52	2.51	2.09	3.58	5.82	7.33	6.40