notesum.ai

Published at May 8

Expanding Sparse Tuning for Low Memory Usage

NeurIPS

Released Date: May 8, 2024

Authors: Shufan Shen¹, Junshu Sun¹, Xiangyang Ji², Qingming Huang¹, Shuhui Wang¹

Aff.: ¹Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS; ²Tsinghua University

Arxiv: https://openreview.net/pdf/9f9ba39d91df45971e3cde402d0940a8f439ff49.pdf

Refer to caption

Method

FGVC

VTAB-1k

CUB-200

NABirds

Oxford

Flowers

Stanford

Dogs

Stanford

Cars

Mean Acc.

Natural

Specialized

Structured

Mean Acc.

Full

87.3

82.7

98.8

89.4

84.5

88.5

75.9

83.4

47.6

69.0

Additional-based methods

MLP-3 [30]

85.1

77.3

97.9

84.9

53.8

79.8

67.8

72.8

30.6

57.1

VPT-Shallow [30]

86.7

78.8

98.4

90.7

68.7

84.6

76.8

79.7

47.0

67.8

VPT-Deep [30]

88.5

84.2

99.0

90.2

83.6

89.1

78.5

82.4

55.0

72.0

Adapter-8 [26]

87.3

84.3

98.4

88.8

68.4

85.5

79.0

84.1

58.5

73.9

Adapter-32 [26]

87.2

84.3

98.5

89.6

68.4

85.6

79.6

84.0

58.3

74.0

SPT-Adapter [22]

89.1

83.3

99.2

91.1

86.2

89.8

82.0

85.8

61.4

76.4

MoSA [60]

89.3

85.7

99.2

91.9

83.4

89.9

79.9

84.0

50.3

71.4

Reparameter-based methods

Linear [30]

85.3

75.9

97.9

86.2

51.3

79.3

68.9

77.2

26.8

57.6

Partial-1 [30]

85.6

77.8

98.2

85.5

66.2

82.6

69.4

78.5

34.2

60.7

Bias [5]

88.4

84.2

98.8

91.2

79.4

88.4

73.3

78.3

44.1

65.2

LoRA-8 [27]

84.9

79.0

98.1

88.1

79.8

86.0

79.5

84.6

60.5

74.9

LoRA-16 [27]

85.6

79.8

98.9

87.6

72.0

84.8

79.8

84.9

60.2

75.0

SPT-LoRA [22]

88.6

83.4

\ul99.5

91.4

87.3

90.1

81.9

85.9

61.3

76.4

SSF [35]

89.5

85.7

99.6

89.6

89.2

90.7

81.6

86.6

59.0

75.7

SNELL-8 (ours)

\ul89.6

\ul86.8

99.3

\ul92.1

89.9

91.5

82.0

85.7

61.6

76.4

SNELL-16 (ours)

89.9

87.0

99.3

92.2

\ul90.3

\ul91.7

\ul82.4

\ul86.1

\ul61.7

\ul76.7

SNELL-32 (ours)

89.9

87.0

\ul99.4

92.0

90.5

91.8

82.7

\ul86.1

61.8

76.9