notesum.ai

Published at December 5

FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression

cs.CV

Released Date: December 5, 2024

Authors: Bo Tong¹, Bokai Lai, Yiyi Zhou, Gen Luo, Yunhang Shen, Ke Li, Xiaoshuai Sun, Rongrong Ji

Aff.: ¹Xiamen University, Key Laboratory of Multimedia Trusted Perception and Efficient Computing, P.R. China

Arxiv: http://arxiv.org/pdf/2412.04317v1

Task

Metric

LLaVA [33]

V1.5-7B

IMP [51]

3.1B

Qwen2-VL [58]

InternVL2 [10]

FlashSloth-HD

3.2B

FlashSloth

3.2B

GQA

Token number

576

729_{$\uparrow 27\%$}

352_{$\downarrow 39\%$}

1699_{$\uparrow 195\%$}

414_{$\downarrow 28\%$}

90_{$\downarrow 84\%$}

TFLOPs

4.41

0.30_{$\downarrow 93\%$}

1.50_{$\downarrow 66\%$}

5.10_{$\uparrow 16\%$}

2.71_{$\downarrow 39\%$}

0.08_{$\downarrow 98\%$}

GPU memory

15.3

8.4_{$\downarrow 45\%$}

9.3_{$\downarrow 39\%$}

7.8_{$\downarrow 49\%$}

7.6_{$\downarrow 50\%$}

Response time

0.11

0.12_{$\uparrow 9\%$}

0.13_{$\uparrow 18\%$}

0.24_{$\uparrow 118\%$}

0.11_0%

0.05_{$\downarrow 55\%$}

Throughput

\uparrow

7.4

9.2_{$\uparrow 24\%$}

5.5_{$\downarrow 26\%$}

3.7_{$\downarrow 50\%$}

7.1_{$\downarrow 4\%$}

12.6_{$\uparrow 70\%$}

TextVQA

Token number

576

729_{$\uparrow 27\%$}

977_{$\uparrow 70\%$}

1668_{$\uparrow 190\%$}

414_{$\downarrow 28\%$}

90_{$\downarrow 84\%$}

TFLOPs

4.76

0.31_{$\downarrow 93\%$}

4.14_{$\downarrow 13\%$}

5.09_{$\uparrow 7\%$}

2.83_{$\downarrow 41\%$}

0.10_{$\downarrow 98\%$}

GPU memory

16.0

8.4_{$\downarrow 48\%$}

40.8_{$\uparrow 155\%$}

8.8_{$\downarrow 45\%$}

8.1_{$\downarrow 49\%$}

7.6_{$\downarrow 53\%$}

Response time

0.11

0.12_{$\uparrow 9\%$}

0.56_{$\uparrow 409\%$}

0.24_{$\uparrow 118\%$}

0.11_$0\%$

0.05_{$\downarrow 55\%$}

Throughput

\uparrow

5.4

7.1_{$\uparrow 31\%$}

1.4_{$\downarrow 74\%$}

3.2_{$\downarrow 41\%$}

5.6_{$\uparrow 4\%$}

9.8_{$\uparrow 81\%$}

MME

Token number

576

729_{$\uparrow 27\%$}

646_{$\uparrow 12\%$}

1478_{$\uparrow 157\%$}

414_{$\downarrow 28\%$}

90_{$\downarrow 84\%$}

TFLOPs

4.43

0.30_{$\downarrow 93\%$}

2.72_{$\downarrow 39\%$}

4.45_$0\%$

2.72_{$\downarrow 39\%$}

0.09_{$\downarrow 98\%$}

GPU memory

15.3

8.4_{$\downarrow 45\%$}

59.2_{$\uparrow 287\%$}

10.0_{$\downarrow 35\%$}

7.8_{$\downarrow 49\%$}

7.6_{$\downarrow 50\%$}

Response time

0.11

0.12_{$\uparrow 9\%$}

0.41_{$\uparrow 273\%$}

0.21_{$\uparrow 91\%$}

0.11_$0\%$

0.05_{$\downarrow 55\%$}

Throughput

\uparrow

5.6

9.1_{$\uparrow 63\%$}

1.8_{$\downarrow 68\%$}

3.7_{$\downarrow 34\%$}

7.3_{$\uparrow 30\%$}

11.6_{$\uparrow 107\%$}

MMB

Token number

576

729_{$\uparrow 27\%$}

385_{$\downarrow 33\%$}

1296_{$\uparrow 125\%$}

414_{$\downarrow 28\%$}

90_{$\downarrow 84\%$}

TFLOPs

4.77

0.31_{$\downarrow 94\%$}

0.84_{$\downarrow 82\%$}

3.98_{$\downarrow 17\%$}

1.98_{$\downarrow 58\%$}

0.09_{$\downarrow 98\%$}

GPU memory

16.4

8.4_{$\downarrow 49\%$}

8.0_{$\downarrow 51\%$}

11.5_{$\downarrow 30\%$}

8.7_{$\downarrow 47\%$}

7.6_{$\downarrow 54\%$}

Response time

0.11

0.12_{$\uparrow 9\%$}

0.09_{$\downarrow 18\%$}

0.19_{$\uparrow 73\%$}

0.11_$0\%$

0.05_{$\downarrow 55\%$}

Throughput

\uparrow

6.1

8.4_{$\uparrow 38\%$}

7.0_{$\uparrow 15\%$}

4.7_{$\downarrow 23\%$}

5.1_{$\downarrow 16\%$}

11.7_{$\uparrow 92\%$}

POPE

Token number

576

729_{$\uparrow 27\%$}

353_{$\downarrow 39\%$}

1666_{$\uparrow 189\%$}

414_{$\downarrow 28\%$}

90_{$\downarrow 84\%$}

TFLOPs

4.40

0.30_{$\downarrow 93\%$}

1.51_{$\downarrow 66\%$}

4.99_{$\uparrow 13\%$}

2.71_{$\downarrow 38\%$}

0.08_{$\downarrow 98\%$}

GPU memory

15.3

8.4_{$\downarrow 45\%$}

7.8_{$\downarrow 49\%$}

7.4_{$\downarrow 52\%$}

7.8_{$\downarrow 49\%$}

7.6_{$\downarrow 50\%$}

Response time

0.11

0.12_{$\uparrow 9\%$}

0.13_{$\uparrow 18\%$}

0.23_{$\uparrow 109\%$}

0.11_$0\%$

0.05_{$\downarrow 55\%$}

Throughput

\uparrow

7.8

9.4_{$\uparrow 21\%$}

5.0_{$\downarrow 36\%$}

3.8_{$\downarrow 51\%$}

7.3_{$\downarrow 6\%$}

13.2_{$\uparrow 69\%$}

Average

Token number

576

729_{$\uparrow 27\%$}

466_{$\downarrow 19\%$}

1561_{$\uparrow 171\%$}

414_{$\downarrow 28\%$}

90_{$\downarrow 84\%$}

TFLOPs

4.55

0.30_{$\downarrow 93\%$}

2.14_{$\downarrow 53\%$}

4.72_{$\uparrow 4\%$}

2.59_{$\downarrow 43\%$}

0.09_{$\downarrow 98\%$}

GPU memory

15.7

8.4_{$\downarrow 46\%$}

30.8_{$\uparrow 96\%$}

9.4_{$\downarrow 40\%$}

8.0_{$\downarrow 49\%$}

7.6_{$\downarrow 52\%$}

Response time

0.11

0.12_{$\uparrow 9\%$}

0.26_{$\uparrow 136\%$}

0.22_{$\uparrow 100\%$}

0.11_$0\%$

0.05_{$\downarrow 55\%$}

Throughput

\uparrow

6.5

8.6_{$\uparrow 32\%$}

4.1_{$\downarrow 37\%$}

3.8_{$\downarrow 42\%$}

6.5_$0\%$

11.8_{$\uparrow 82\%$}