notesum.ai

Published at October 20

Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs

eess.SY

cs.AI

cs.LG

cs.SY

Released Date: October 20, 2024

Authors: Xin Zhou¹, Ping Nie², Yiwen Guo³, Haojie Wei², Zhanqiu Zhang², Pasquale Minervini⁴, Ruotian Ma⁵, Tao Gui⁵, Qi Zhang⁶, Xuanjing Huang⁶

Aff.: ¹School of Computer Science, Fudan University, Shanghai, China; ²LightSpeed Studios, Tencent; ³Independent Researcher; ⁴School of Informatics and ELLIS, University of Edinburgh; ⁵Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China; ⁶Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China

Arxiv: https://arxiv.org/abs/2410.15438v1

Model	Method	PopQA	RGBqa
Mixtral-8x7B	Random Guess	42.01	40.28
	Knowledge (50-Shot)	55.87	55.15
	Knowledge (Full-Set)	56.77	59.84
Mixtral-8x22B	Random Guess	47.39	47.18
	Knowledge (50-Shot)	64.78	73.31
	Knowledge (Full-Set)	65.83	75.40