notesum.ai

Published at November 11

Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training

cs.LG

cs.AI

cs.CL

Released Date: November 11, 2024

Authors: Elia Cunegatti¹, Leonardo Lucio Custode¹, Giovanni Iacca¹

Aff.: ¹Department of Information Engineering and Computer Science, University of Trento, Italy

Arxiv: http://arxiv.org/abs/2411.07066v1

Refer to caption

Dataset	Model
Dataset	Phi-2.7B	LLama1 7B	LLama2 7B	Mistral-7B	OPT-6.7B
WikiText2	24.5 $\pm$ 0.6	09.5 $\pm$ 0.1	9.3 $\pm$ 0.0	10.1 $\pm$ 0.1	16.2 $\pm$ 0.1
C4	27.0 $\pm$ 0.2	11.9 $\pm$ 0.1	11.9 $\pm$ 0.0	13.8 $\pm$ 0.1	19.1 $\pm$ 0.1
PTB	42.3 $\pm$ 0.7	17.1 $\pm$ 0.3	65.3 $\pm$ 0.8	74.9 $\pm$ 0.2	25.1 $\pm$ 0.1