notesum.ai

Published at October 24

Dynamic Vocabulary Pruning in Early-Exit LLMs

cs.RO

cs.AI

Released Date: October 24, 2024

Authors: Jort Vincenti¹, Karim Abdel Sadek¹, Joan Velja¹, Matteo Nulli¹, Metod Jazbec¹

Aff.: ¹University of Amsterdam

Arxiv: https://arxiv.org/abs/2410.18952v1

Refer to caption

Dataset	Conf. $\lambda$	Method	Score ( $\uparrow$ )	FLOPs/Token ( $\downarrow$ )	Avg. Exit ( $\downarrow$ )	Conf. Time (s) ( $\downarrow$ )
SQuAD [13]	$0.6$	CALM	87.5	2.21 $\times$ $10^{8}$	2.4	44.5
	$0.6$	+ DVP (ours)	87.4	1.97 $\times$ $\mathbf{10^{8}}$	2.4	40.8
	$\approx 0.99$	CALM	90.6	13.91 $\times$ $10^{8}$	20.9	499.9
	$\approx 0.99$	+ DVP (ours)	90.6	1.99 $\times$ $\mathbf{10^{8}}$	20.8	413.1
SamSum [6]	$0.6$	CALM	33.8	4.21 $\times$ $10^{8}$	5.5	90.0
	$0.6$	+ DVP (ours)	33.7	2.01 $\times$ $\mathbf{10^{8}}$	5.4	81.0
	$\approx 0.99$	CALM	43.1	11.13 $\times$ $10^{8}$	16.5	162.0
	$\approx 0.99$	+ DVP (ours)	43.1	2.12 $\times$ $\mathbf{10^{8}}$	16.4	136.0