notesum.ai

Published at November 1

Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM

cs.SD

cs.AI

cs.CL

eess.AS

Released Date: November 1, 2024

Authors: Xiong Wang¹, Yangze Li², Chaoyou Fu³, Lei Xie², Ke Li¹, Xing Sun¹, Long Ma¹

Aff.: ¹Tencent Youtu Lab; ²Audio, Speech and Language Processing Group (ASLP@NPU); ³Nanjing University

Arxiv: http://arxiv.org/abs/2411.00774v1

Refer to caption

Model	Modality	Web Q.	LlaMA Q.	Audio Trivia QA
SpeechGPT(7B) [27]	Audio&Text	6.5	21.6	14.8
Spectron(1B) [17]	Audio&Text	6.1	22.9	-
Moshi(7B) [7]	Audio&Text	26.6	62.3	22.8
Freeze-Omni(7B)	Audio&Text	44.73	72	53.88
Helium [7]	Text Only	32.3	75	56.4
Qwen2-7B-Instruct	Text Only	45.13	77.67	63.93