notesum.ai

Published at October 21

Pruning Foundation Models for High Accuracy without Retraining

cs.CR

cs.AI

Released Date: October 21, 2024

Authors: Pu Zhao¹, Fei Sun², Xuan Shen¹, Pinrui Yu¹, Zhenglun Kong¹, Yanzhi Wang¹, Xue Lin¹

Aff.: ¹Northeastern University; ²Meta

Arxiv: https://arxiv.org/abs/2410.15567v1

Model & Setting

Datasets

Origin

Unstructured 50%

2:4 sparsity (50% sparsity)

\mathfrak{S}\mathfrak{S}

(SparseGPT)

\mathfrak{S}\mathfrak{M}

(ours)

\mathfrak{S}\mathfrak{S}

(SparseGPT)

\mathfrak{S}\mathfrak{M}

(ours)

\mathfrak{M}\mathfrak{S}

(ours)

\mathfrak{M}\mathfrak{M}

(ours)

LLaMA2-7B S=2048

wikitext2

5.472

7.052

7.018

10.85

10.15

10.7

10.14

7.263

9.305

9.204

13.65

12.48

13.38

12.47

LLaMA2-7B S=all

wikitext2

5.472

7.045

7.019

10.92

10.37

10.6

10.38

7.263

9.36

9.247

13.62

12.762

13.31

12.759

LLaMA2-13B S=2048

wikitext2

4.884

6.028

6.001

8.76

8.219

8.644

8.224

6.727

8.275

8.21

11.4

10.71

11.28

10.699

LLaMA2-13B S=all

wikitext2

4.884

6.082

6.03

8.732

8.239

8.65

8.225

6.727

8.374

8.269

11.36

10.796

11.23

10.789

LLaMA2-70B S=all

wikitext2

3.319

4.509

4.142

5.698

4.278

4.353

4.278

5.709

6.932

6.528

8.154

6.683

6.84

6.683

OPT-2.7B S=512

wikitext2

12.47

13.43

13.29

17.13

16.74

16.89

16.68

14.34

15.8

15.66

19.34

18.7

19.07

18.69

OPT-6.7B S=2048

wikitext2

10.86

11.64

11.57

14.16

13.73

14.19

13.72

12.71

13.81

13.77

16.42

15.86

16.34

15.85

OPT-30B S=all

wikitext2

9.558

9.926

9.824

10.9

10.7

10.88

10.7

11.44

12.12

11.98

13.16

12.93

13.13

12.93

BLOOM-1.7B S=512

wikitext2

15.39

19.1

18.67

23.7

22.91

24.01

22.86

19.49

22.53

22.06

27.02

25.95

27.42

25.88

BLOOM-3B S=2048

wikitext2

13.48

15.99

15.56

18.87

18.6

18.8

18.57

17.48

19.76

19.3

22.81

22.22

22.77

22.2

BLOOM-7.1B S=2048

wikitext2

11.37

12.86

14.87

14.57

14.82

14.57

15.2

16.71

16.59

18.79

18.47

18.74

18.47