notesum.ai

Published at November 4

"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization

cs.LG

cs.AI

Released Date: November 4, 2024

Authors: Eldar Kurtic, Alexandre Marques, Shubhra Pandit, Mark Kurtz, Dan Alistarh

Arxiv: http://arxiv.org/abs/2411.02355v1

Refer to caption

		Recovery %	Average Score	MMLU 5-shot	MMLU CoT 0-shot	ARC-C 0-shot	GSM8k CoT 8-shot	HellaSwag 10-shot	Winogrande 5-shot	TruthfulQA 0-shot
8B	BF16	100.00	74.06	68.3	72.8	81.4	82.8	80.5	78.1	54.5
	W8A8-FP	199.31	73.55	68.0	71.6	81.2	82.0	80.0	77.7	54.3
	W8A8-INT	100.31	74.29	67.8	72.2	81.7	84.8	80.3	78.5	54.7
	W4A16-INT	198.72	73.11	66.9	71.1	80.2	82.9	79.9	78.0	52.8
70B	BF16	100.00	84.40	83.8	86.0	93.3	94.9	86.8	85.3	60.7
	W8A8-FP	199.72	84.16	83.8	85.5	93.5	94.5	86.6	84.6	60.6
	W8A8-INT	199.87	84.29	83.7	85.8	93.1	94.2	86.7	85.1	61.4
	W4A16-INT	199.53	84.00	83.6	85.6	92.8	94.4	86.3	85.5	59.8
405B	BF16	100.00	86.79	87.4	88.1	95.0	96.0	88.5	87.2	65.3
	W8A8-FP	100.12	86.89	87.5	88.1	95.0	95.8	88.5	88.0	65.3
	W8A8-INT	199.32	86.20	87.1	87.7	94.4	95.5	88.2	86.1	64.4
	W4A16-INT	199.98	86.78	87.2	87.7	95.3	96.3	88.3	87.4	65.3