notesum.ai

Published at November 21

DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization

cs.CL

Released Date: November 21, 2024

Authors: Hexuan Deng¹, Wenxiang Jiao¹, Xuebo Liu¹, Min Zhang¹, Zhaopeng Tu¹

Aff.: ¹Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China

Arxiv: http://arxiv.org/abs/2411.14055v1

Refer to caption

Method	From	To	PPL $\downarrow$	Down. $\uparrow$
Sheared Llama	7B	1.3B	10.05	34.89
ReSheared	7B	1.3B	10.42	34.85
DRPruning	7B	1.3B	9.83	35.60
Sheared Llama	7B	2.7B	7.64	39.75
ReSheared	7B	2.7B	7.83	39.98
DRPruning	7B	2.7B	7.40	40.18