notesum.ai

Published at October 31

BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments

cs.CL

cs.AI

cs.CV

cs.LG

Released Date: October 31, 2024

Authors: Xinghao Wang¹, Pengyu Wang¹, Bo Wang¹, Dong Zhang¹, Yunhua Zhou², Xipeng Qiu¹

Aff.: ¹Fudan University; ²Shanghai Artificial Intelligence Laboratory

Arxiv: http://arxiv.org/abs/2410.23918v1

Model	Memory	Method	Wiki2	ARC-e	ARC-c	PIQA	HellaS.	WinoG.	LAMBADA	Avg.
Model	(MB)	Method	( $\downarrow$ )	( $\uparrow$ )	( $\uparrow$ )	( $\uparrow$ )	( $\uparrow$ )	( $\uparrow$ )	( $\uparrow$ )	( $\uparrow$ )
7B	12852	FP 16	5.47	74.5_±0.9	46.2_±1.5	79.1_±0.9	76.0 ${}_{\pm}0.4$	69.1_±1.3	73.9_±0.6	69.8_±0.9
\cdashline2-12	2050_(84%)	GPTQ_w2	2.8e4	26.5_±0.9	27.6_±1.3	48.4_±1.2	25.9 ${}_{\pm}0.4$	50.3_±1.4	0.0_±0.0	29.8_±0.9
		AWQ_w2	1.8e5	26.3_±0.9	26.7_±1.3	50.9_±1.2	26.5 ${}_{\pm}0.4$	49.3_±1.4	0.0_±0.0	30.0_±0.9
		BitStack	29.93	32.3_±1.0	25.6_±1.3	62.4_±1.1	42.8 ${}_{\pm}0.5$	53.6_±1.4	24.7_±0.6	40.2_±1.0
\cdashline2-12	2238_(83%)	GPTQ_w2g128	156.37	28.2_±0.9	27.1_±1.3	51.7_±1.2	28.0 ${}_{\pm}0.4$	51.1_±1.4	0.3_±0.1	31.1_±0.9
		AWQ_w2g128	2.3e5	25.8_±0.9	26.7_±1.3	50.2_±1.2	26.1 ${}_{\pm}0.4$	49.8_±1.4	0.0_±0.0	29.8_±0.9
		BitStack	12.49	51.8_±1.0	30.1_±1.3	71.1_±1.1	53.0 ${}_{\pm}0.5$	61.1_±1.4	53.3_±0.7	53.4_±1.0
\cdashline2-12	2822_(78%)	GPTQ_w3	9.38	58.1_±1.0	34.0_±1.4	71.9_±1.0	61.7 ${}_{\pm}0.5$	60.6_±1.4	53.3_±0.7	56.6_±1.0
		AWQ_w3	14.33	52.7_±1.0	33.0_±1.4	68.3_±1.1	56.3 ${}_{\pm}0.5$	59.3_±1.4	36.3_±0.7	51.0_±1.0
		BitStack	7.45	62.5_±1.0	37.5_±1.4	74.8_±1.0	67.0 ${}_{\pm}0.5$	66.5_±1.3	68.5_±0.6	62.8_±1.0
\cdashline2-12	3010_(77%)	GPTQ_w3g128	922.54	26.3_±0.9	25.3_±1.3	52.4_±1.2	27.4 ${}_{\pm}0.4$	49.0_±1.4	0.1_±0.0	30.1_±0.9
		AWQ_w3g128	6.14	70.2_±0.9	43.7_±1.4	78.0_±1.0	73.9 ${}_{\pm}0.4$	67.6_±1.3	71.4_±0.6	67.5_±1.0
		BitStack	7.10	63.8_±1.0	38.2_±1.4	76.0_±1.0	68.4 ${}_{\pm}0.5$	65.9_±1.3	70.7_±0.6	63.8_±1.0
\cdashline2-12	3594_(72%)	GPTQ_w4	5.91	71.8_±0.9	43.7_±1.4	77.7_±1.0	74.5 ${}_{\pm}0.4$	68.7_±1.3	71.1_±0.6	67.9_±1.0
		AWQ_w4	5.81	70.9_±0.9	44.5_±1.5	78.5_±1.0	74.8 ${}_{\pm}0.4$	69.2_±1.3	71.5_±0.6	68.2_±1.0
		BitStack	6.36	67.0_±1.0	41.4_±1.4	77.1_±1.0	71.4 ${}_{\pm}0.5$	69.5_±1.3	73.1_±0.6	66.6_±1.0
\cdashline2-12	3782_(71%)	GPTQ_w4g128	5.73	73.6_±0.9	45.3_±1.5	78.7_±1.0	75.4 ${}_{\pm}0.4$	67.6_±1.3	72.7_±0.6	68.9_±0.9
		AWQ_w4g128	5.61	73.3_±0.9	45.2_±1.5	78.6_±1.0	75.2 ${}_{\pm}0.4$	68.7_±1.3	72.7_±0.6	68.9_±0.9
		BitStack	6.27	67.8_±1.0	43.3_±1.4	77.2_±1.0	72.2 ${}_{\pm}0.4$	68.6_±1.3	73.9_±0.6	67.2_±1.0
13B	24825	FP 16	4.88	77.4_±0.9	49.1_±1.5	80.5_±0.9	79.4 ${}_{\pm}0.4$	72.2_±1.3	76.8_±0.6	72.6_±0.9
\cdashline2-12	3659_(85%)	GPTQ_w2	1.2e4	26.4_±0.9	28.2_±1.3	50.2_±1.2	26.3 ${}_{\pm}0.4$	48.4_±1.4	0.0_±0.0	29.9_±0.9
		AWQ_w2	9.6e4	27.3_±0.9	28.0_±1.3	49.9_±1.2	26.0 ${}_{\pm}0.4$	50.4_±1.4	0.0_±0.0	30.3_±0.9
		BitStack	68.64	38.1_±1.0	23.5_±1.2	57.3_±1.2	32.2 ${}_{\pm}0.5$	51.6_±1.4	14.0_±0.5	36.1_±1.0
\cdashline2-12	4029_(84%)	GPTQ_w2g128	3.9e3	26.2_±0.9	28.8_±1.3	50.7_±1.2	26.9 ${}_{\pm}0.4$	48.6_±1.4	0.1_±0.0	30.2_±0.9
		AWQ_w2g128	1.2e5	26.9_±0.9	27.5_±1.3	50.0_±1.2	26.1 ${}_{\pm}0.4$	50.8_±1.4	0.0_±0.0	30.2_±0.9
		BitStack	9.26	64.5_±1.0	34.2_±1.4	73.0_±1.0	60.9 ${}_{\pm}0.5$	64.9_±1.3	65.3_±0.7	60.5_±1.0
\cdashline2-12	5171_(79%)	GPTQ_w3	6.20	68.2_±1.0	42.8_±1.4	77.1_±1.0	71.4 ${}_{\pm}0.5$	67.6_±1.3	63.1_±0.7	65.0_±1.0
		AWQ_w3	6.46	71.1_±0.9	44.4_±1.5	77.6_±1.0	71.2 ${}_{\pm}0.5$	66.8_±1.3	61.9_±0.7	65.5_±1.0
		BitStack	6.32	74.4_±0.9	45.1_±1.5	77.1_±1.0	71.9 ${}_{\pm}0.4$	69.2_±1.3	74.8_±0.6	68.8_±0.9
\cdashline2-12	5541_(78%)	GPTQ_w3g128	5.85	73.4_±0.9	45.2_±1.5	78.2_±1.0	74.4 ${}_{\pm}0.4$	68.0_±1.3	67.6_±0.7	67.8_±1.0
		AWQ_w3g128	5.29	75.3_±0.9	48.5_±1.5	79.4_±0.9	77.1 ${}_{\pm}0.4$	70.8_±1.3	75.1_±0.6	71.0_±0.9
		BitStack	6.04	74.4_±0.9	46.2_±1.5	77.9_±1.0	72.6 ${}_{\pm}0.4$	70.6_±1.3	76.6_±0.6	69.7_±0.9
\cdashline2-12	6684_(73%)	GPTQ_w4	5.09	75.8_±0.9	48.0_±1.5	79.6_±0.9	77.8 ${}_{\pm}0.4$	72.4_±1.3	74.5_±0.6	71.4_±0.9
		AWQ_w4	5.07	78.2_±0.8	49.7_±1.5	80.4_±0.9	78.6 ${}_{\pm}0.4$	71.6_±1.3	76.1_±0.6	72.4_±0.9
		BitStack	5.53	76.7_±0.9	48.4_±1.5	79.0_±1.0	75.2 ${}_{\pm}0.4$	71.7_±1.3	77.4_±0.6	71.4_±0.9
\cdashline2-12	7054_(72%)	GPTQ_w4g128	4.97	76.4_±0.9	49.2_±1.5	79.9_±0.9	78.8 ${}_{\pm}0.4$	71.7_±1.3	76.0_±0.6	72.0_±0.9
		AWQ_w4g128	4.97	77.1_±0.9	48.5_±1.5	80.4_±0.9	78.8 ${}_{\pm}0.4$	73.1_±1.2	76.8_±0.6	72.5_±0.9
		BitStack	5.47	76.5_±0.9	48.0_±1.5	79.0_±1.0	75.7 ${}_{\pm}0.4$	71.7_±1.3	77.8_±0.6	71.4_±0.9
70B	131562	FP 16	3.32	81.1_±0.8	57.3_±1.4	82.7_±0.9	83.8 ${}_{\pm}0.4$	78.0_±1.2	79.6_±0.6	77.1_±0.9
\cdashline2-12	17348_(87%)	GPTQ_w2	152.31	26.8_±0.9	26.0_±1.3	49.0_±1.2	26.1 ${}_{\pm}0.4$	49.8_±1.4	0.0_±0.0	29.6_±0.9
		AWQ_w2	8.0e4	25.8_±0.9	28.8_±1.3	50.1_±1.2	25.7 ${}_{\pm}0.4$	48.3_±1.4	0.0_±0.0	29.8_±0.9
		BitStack	9.41	67.8_±1.0	42.1_±1.4	75.9_±1.0	65.1 ${}_{\pm}0.5$	67.7_±1.3	65.7_±0.7	64.1_±1.0
\cdashline2-12	19363_(85%)	GPTQ_w2g128	7.79	53.0_±1.0	32.0_±1.4	66.9_±1.1	51.1 ${}_{\pm}0.5$	60.2_±1.4	34.8_±0.7	49.7_±1.0
		AWQ_w2g128	7.2e4	26.0_±0.9	28.9_±1.3	49.8_±1.2	25.7 ${}_{\pm}0.4$	51.0_±1.4	0.0_±0.0	30.2_±0.9
		BitStack	5.30	74.5_±0.9	50.0_±1.5	79.7_±0.9	75.1 ${}_{\pm}0.4$	74.4_±1.2	79.3_±0.6	72.2_±0.9
\cdashline2-12	25508_(81%)	GPTQ_w3	4.49	75.9_±0.9	52.1_±1.5	80.7_±0.9	79.2 ${}_{\pm}0.4$	75.3_±1.2	74.3_±0.6	72.9_±0.9
		AWQ_w3	4.30	79.8_±0.8	55.4_±1.5	81.4_±0.9	81.2 ${}_{\pm}0.4$	73.6_±1.2	73.1_±0.6	74.1_±0.9
		BitStack	4.33	78.9_±0.8	54.9_±1.5	81.7_±0.9	79.9 ${}_{\pm}0.4$	76.6_±1.2	80.1_±0.6	75.3_±0.9
\cdashline2-12	27523_(79%)	GPTQ_w3g128	55.43	27.8_±0.9	27.4_±1.3	50.9_±1.2	29.8 ${}_{\pm}0.5$	48.9_±1.4	9.5_±0.4	32.4_±0.9
		AWQ_w3g128	3.74	79.0_±0.8	56.7_±1.4	82.8_±0.9	82.3 ${}_{\pm}0.4$	76.6_±1.2	79.3_±0.6	76.1_±0.9
		BitStack	4.07	79.8_±0.8	55.4_±1.5	82.4_±0.9	80.7 ${}_{\pm}0.4$	77.3_±1.2	81.6_±0.5	76.2_±0.9
\cdashline2-12	33668_(74%)	GPTQ_w4	3.59	79.3_±0.8	54.9_±1.5	82.2_±0.9	82.8 ${}_{\pm}0.4$	77.2_±1.2	79.1_±0.6	75.9_±0.9
		AWQ_w4	3.48	80.6_±0.8	57.9_±1.4	82.8_±0.9	83.2 ${}_{\pm}0.4$	76.5_±1.2	78.8_±0.6	76.6_±0.9
		BitStack	3.76	79.3_±0.8	57.4_±1.4	82.4_±0.9	81.8 ${}_{\pm}0.4$	77.9_±1.2	81.0_±0.5	76.6_±0.9
\cdashline2-12	35683_(73%)	GPTQ_w4g128	3.42	81.3_±0.8	57.8_±1.4	83.0_±0.9	83.6 ${}_{\pm}0.4$	76.8_±1.2	79.4_±0.6	77.0_±0.9
		AWQ_w4g128	3.41	80.3_±0.8	56.7_±1.4	83.1_±0.9	83.4 ${}_{\pm}0.4$	78.1_±1.2	79.6_±0.6	76.9_±0.9
		BitStack	3.71	79.7_±0.8	57.1_±1.4	82.2_±0.9	82.1 ${}_{\pm}0.4$	77.9_±1.2	81.7_±0.5	76.8_±0.9