notesum.ai

Published at December 5

p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay

cs.CV

cs.CL

Released Date: December 5, 2024

Authors: Jun Zhang¹, Desen Meng, Ji Qi, Zhenpeng Huang, Tao Wu, Limin Wang

Aff.: ¹State Key Laboratory for Novel Software Technology, Nanjing University

Arxiv: http://arxiv.org/pdf/2412.04449v1

Model

Params

Inference

TFLOPs

Inference KV

cache storage

SEED

RWQA

MME

MMB

POPE

GQA

AI2D

AVG

LLaVA-v1.5

8.38

100%

66.2

55.6

1506.8

64.1

85.9

61.9

55.2

66.3

p-MoD-LLaVA-v1.5

4.92

{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(-41.3\%)}

53.8%

66.5

55.7

1482.8

65.4

85.5

62.2

56.2

66.5

LLaVA-NeXT

39.46

100%

68.9

57.6

1519.3

67.5

87.2

63.5

64.0

69.2

p-MoD-LLaVA-NeXT

21.94

{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(-44.4\%)}

53.8%

69.0

57.6

1495.5

67.3

86.8

63.3

65.1

69.1