notesum.ai

Published at November 26

Attamba: Attending To Multi-Token States

cs.LG

cs.CL

Released Date: November 26, 2024

Authors: Yash Akhauri¹, Safeen Huda², Mohamed S. Abdelfattah¹

Aff.: ¹1; ²2

Arxiv: http://arxiv.org/abs/2411.17685v1

Refer to caption

P	Attamba	IsoParam	IsoFLOP	IsoKV
4	512	512	160	128
8	512	512	104	64