notesum.ai

Published at November 26

Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens

cs.LG

cs.CL

Released Date: November 26, 2024

Authors: Xu Ouyang¹, Tao Ge², Thomas Hartvigsen¹, Zhisong Zhang², Haitao Mi², Dong Yu²

Aff.: ¹University of Virginia; ²Tencent AI Lab

Arxiv: http://arxiv.org/abs/2411.17691v1

Refer to caption

Model Size	$\Delta_{q}$ Loss = 0.2			$\Delta_{q}$ Loss = 0.3			$\Delta_{q}$ Loss = 0.4			$\Delta_{q}$ Loss = 0.5
Model Size	2 bits	3 bits	4 bits	2 bits	3 bits	4 bits	2 bits	3 bits	4 bits	2 bits	3 bits	4 bits
1B	0.0011	0.1089	1.4424	0.0025	0.1990	2.6786	0.0043	0.3051	4.1556	0.0066	0.4251	5.8422
7B	0.0026	0.3038	4.5066	0.0057	0.5550	8.3689	0.0099	0.8512	12.9836	0.0152	1.1860	18.2531
70B	0.0071	1.0228	17.3499	0.0154	1.8687	32.2192	0.0267	2.8659	49.9854	0.0409	3.9932	70.2723
405B	0.0151	2.5807	48.4861	0.0328	4.7151	90.0398	0.0567	7.2311	139.6892	0.0868	10.0754	196.3829