notesum.ai

Published at November 6

Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models

cs.CL

cs.AI

cs.LG

Released Date: November 6, 2024

Authors: Zhijian Zhuo¹, Ya Wang², Yutao Zeng², Xiaoqing Li³, Xun Zhou², Jinwen Ma¹

Aff.: ¹School of Mathematical Sciences, Peking University; ²Seed-Foundation-Model, ByteDance; ³Capital University of Economics and Business

Arxiv: http://arxiv.org/abs/2411.03884v1

Refer to caption

	Loss $\downarrow$	PPL $\downarrow$	ARC-E	ARC-C	HellaSwag	PIQA	SciQ	Winograde	Avg. $\uparrow$
SwiGLU	2.19	3.22	56.61	27.47	49.23	68.61	86.10	56.83	57.47
GELU	2.20	3.24	55.43	27.73	48.42	68.12	87.40	54.78	56.98
ReLU	2.21	3.26	55.68	28.50	48.59	68.39	87.10	54.85	57.18
PolyReLU	2.17	3.18	57.53	27.99	50.19	70.29	87.60	55.72	58.22
PolyNorm	2.17	3.17	59.68	29.01	50.86	69.15	87.20	56.20	58.68