notesum.ai

Published at December 5

Densing Law of LLMs

cs.AI

cs.CL

Released Date: December 5, 2024

Authors: Chaojun Xiao¹, Jie Cai², Weilin Zhao¹, Guoyang Zeng², Xu Han¹, Zhiyuan Liu¹, Maosong Sun¹

Aff.: ¹Tsinghua University; ²ModelBest Inc.

Arxiv: http://arxiv.org/pdf/2412.04315v1

Refer to caption

Name	# Para	BS	$n_{layer}$	$d$	$d_{ffn}$	$d_{head}$	$n_{head}$	$n_{kv}$
0.005B	5,247,232	32	8	256	640	64	4	1
0.03B	31,470,080	32	12	512	1,280	64	8	2
0.1B	106,196,736	64	18	768	1,920	64	12	3
0.2B	245,416,960	128	24	1,024	2,560	64	16	2
0.4B	476,852,480	256	30	1,280	3,200	64	20	2
0.8B	828,225,024	512	36	1,536	3,840	64	24	3