notesum.ai

Published at November 8

MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization

cs.AR

cs.AI

cs.LG

Released Date: November 8, 2024

Authors: Akshat Ramachandran¹, Souvik Kundu², Tushar Krishna¹

Aff.: ¹Georgia Institute of Technology; ²Intel Labs

Arxiv: http://arxiv.org/abs/2411.05282v1

[Uncaptioned image]

		OPT (Zhang et al., 2022)		Llama-2 (Touvron et al., 2023)			Llama-3 (Meta, 2024)		Mixtral (Jiang et al., 2024)	Phi-3 (Abdin et al., 2024)
Method	W/A	6.7B	175B	7B	13B	70B	8B	70B	8x7B	3.8B	14B
Baseline	16/16	10.86	8.34	5.47	4.83	3.31	6.13	2.85	3.84	6.33	4.31
OliVe (Guo et al., 2022)	4/16	12.20	9.09	11.52	9.34	7.23	10.29	5.65	6.19	8.57	7.81
GOBO (Zadeh et al., 2020)	4/16	10.97	8.71	5.79	5.03	3.45	7.11	3.53	4.22	6.64	4.78
GPTQ (Frantar et al., 2022)	4/16	11.12	9.09	6.23	5.58	4.28	8.12	3.75	4.68	7.17	5.13
AWQ (Lin et al., 2024a)	4/16	10.97	8.74	5.82	5.19	4.08	7.96	3.58	4.36	6.72	4.99
OmniQuant (Shao et al., 2023)	4/16	10.96	8.72	5.74	5.02	3.47	7.09	3.46	4.19	6.67	4.82
MicroScopiQ (Ours)	4/16	10.97	8.69	5.74	5.07	3.42	7.12	3.37	4.25	6.61	4.73
OliVe (Guo et al., 2023)	4/4	55.44	14.17	19.28	14.96	13.59	27.65	9.34	23.53	17.63	15.29
OmniQuant (Shao et al., 2023)	4/4	11.61	9.88	11.47	8.32	5.41	10.21	5.30	5.98	8.21	6.40
MicroScopiQ (Ours)	4/4	11.72	9.75	11.47	8.29	5.48	9.53	5.23	6.01	7.54	6.37
MicroScopiQ ${}_{\text{act\_out}}$ (Ours)	4/4	11.12	9.07	10.01	6.86	4.96	9.11	4.58	5.81	7.10	5.83
OmniQuant (Shao et al., 2023)	2/16	11.61	9.66	9.62	7.56	6.11	9.13	6.17	6.02	7.09	6.28
MicroScopiQ (Ours)	2/16	11.68	9.53	8.43	7.57	6.01	8.97	5.91	6.02	7.16	6.03
OmniQuant (Shao et al., 2023)	2/8	11.99	10.23	9.62	8.92	6.83	9.39	6.59	6.29	7.95	7.37
MicroScopiQ (Ours)	2/8	11.94	9.98	9.06	8.06	6.59	9.17	6.53	6.27	7.98	7.02
MicroScopiQ ${}_{\text{act\_out}}$ (Ours)	2/8	11.71	9.91	9.06	7.73	6.31	9.08	6.13	6.19	7.54	6.73