notesum.ai

Published at November 16

SAM Decoding: Speculative Decoding via Suffix Automaton

cs.CL

cs.AI

Released Date: November 16, 2024

Authors: Yuxuan Hu, Ke Wang, Jing Zhang, Cuiping Li, Hong Chen

Arxiv: http://arxiv.org/abs/2411.10666v1

Model	Method	HumanEval			MBPP			HAGRID
Model	Method	#MAT	Tokens/s	Speedup	#MAT	Tokens/s	Speedup	#MAT	Tokens/s	Speedup
Vicuna-7B	PLD	1.65	59.04	$1.52\times$	1.42	50.62	$1.27\times$	2.03	44.11	$1.29\times$
	Token Recycling	2.78	75.44	$1.94\times$	2.83	79.22	$1.99\times$	2.88	66.17	$1.93\times$
	SAM-Decoding[T]	2.94	95.08	$2.45\times$	2.87	94.50	$2.37\times$	3.23	87.93	$2.57\times$
	EAGLE	4.10	103.39	$2.66\times$	4.17	119.31	$3.00\times$	3.44	72.14	$2.11\times$
	SAM-Decoding[E]	4.03	110.16	$2.83\times$	3.85	114.92	$2.87\times$	4.05	93.17	$2.72\times$
	EAGLE2	5.12	128.19	$3.30\times$	5.29	135.24	$3.40\times$	4.15	82.61	$2.41\times$
	SAM-Decoding[E2]	4.79	127.03	$3.27\times$	4.50	117.06	$2.94\times$	4.75	96.60	$2.81\times$