notesum.ai

Published at October 23

cs.LG

cs.AI

cs.CV

Released Date: October 23, 2024

Authors: Guangyuan Shi¹, Zexin Lu¹, Xiaoyu Dong¹, Wenlong Zhang¹, Xuanyu Zhang², Yujie Feng¹, Xiao-Ming Wu¹

Aff.: ¹Department of Computing, The Hong Kong Polytechnic University, Hong Kong S.A.R., China; ²Du Xiaoman Financial, China

	ATT ( $W_{q}$ , $W_{k}$ , $W_{v}$ , $W_{o}$ )	ATT2 ( $W_{q}$ , $W_{k}$ , $W_{v}$ )	FFN ( $W_{\text{up}}$ , $W_{\text{down}}$ , $W_{\text{gate}}$ )	ALL (LoRA)
MMLU $\uparrow$	42.03	42.65	43.06	43.18
Vicuna $\uparrow$	5.63	5.54	5.69	5.78
MT-Bench $\uparrow$	3.82	3.80	3.92	3.98