notesum.ai

Published at November 18

BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

cs.AR

Released Date: November 18, 2024

Authors: Yuzong Chen¹, Ahmed F. AbouElhamayed¹, Xilai Dai¹, Yang Wang², Marta Andronic³, George A. Constantinides³, Mohamed S. Abdelfattah¹

Aff.: ¹Computer Systems Lab, Cornell University; ²Systems and Networking Research Group, Microsoft Research; ³Department of Electrical and Electronic Engineering, Imperial College London

Arxiv: http://arxiv.org/abs/2411.11745v1

Precision	Datatype	OPT-1.3B			Phi-2B			Yi-6B			Llama-2-7B			Llama-2-13B			Llama-3-8B			Mean $\Delta$ Acc
Precision	Datatype	Hella	Wino	Piqa	Hella	Wino	Piqa	Hella	Wino	Piqa	Hella	Wino	Piqa	Hella	Wino	Piqa	Hella	Wino	Piqa	Mean $\Delta$ Acc
16-bit	FP16	53.72	59.43	72.41	73.74	75.77	79.22	74.96	70.72	78.78	75.98	69.06	79.11	79.39	72.38	80.5	79.18	72.85	80.74	0
4-bit	INT4-Asym	52.31	59.35	71.05	72.29	75.14	78.4	73.91	70.51	77.64	75.29	68.74	78.22	78.76	72.45	80.2	78.07	73.24	79.76	-0.71
4-bit	BitMoD	53.03	59.12	71.49	72.51	77.58	79.48	73.98	70.09	78.35	75.43	68.19	78.45	78.41	72.14	80.42	78.49	73.09	79.98	-0.42
3-bit	INT3-Asym	38.98	55.01	64.25	67.75	71.74	77.48	71.3	67.32	76.71	71.87	66.46	76.66	76.58	69.61	78.94	68.56	66.61	75.03	-4.84
3-bit	BitMoD	49.16	58.09	68.88	70.16	75.22	78.18	70.72	67.72	76.28	72.68	66.22	77.53	76.79	72.37	79.22	73.56	70.32	77.91	-2.61