notesum.ai

Published at December 4

Continual Low-Rank Scaled Dot-product Attention

cs.CV

cs.LG

Released Date: December 4, 2024

Authors: Ginés Carreto Picón, Illia Oleksiienko¹, Lukas Hedegaard¹, Arian Bakhtiarnia¹, Alexandros Iosifidis¹

Aff.: ¹Aarhus University, Denmark

Arxiv: http://arxiv.org/pdf/2412.03214v1

Refer to caption

Model	Layers	Acc. (%)	FLOPs (M)	Rel. FLOPs
Att	1	$93.12\pm 5.44$	11.1	$\times$ 1
Att	2	$94.32\pm 4.31$	22.19	$\times$ 1
$\text{Att}_{\text{Co}}$	1	$93.16\pm 5.39$	0.09	$\times$ 123.3
$\text{Att}_{\text{Co}}$	2	$94.52\pm 4.05$	0.39	$\times$ 56.90
$\text{Att}_{\text{Ny}}$ -4	1	93.54 $\pm 4.26$	0.42	$\times$ 26.43
$\text{Att}_{\text{Ny}}$ -4	2	93.92 $\pm 4.43$	0.85	$\times$ 26.11
$\text{Att}_{\text{Ny}}^{\text{Fix}}$ -4	1	$93.24\pm 5.30$	0.42	$\times$ 26.43
$\text{Att}_{\text{Ny}}^{\text{Fix}}$ -4	2	$93.76\pm 5.14$	0.84	$\times$ 26.42
$\text{Att}_{\text{CoNy}}^{\text{Cont}}$ -4	1	$92.92\pm 5.43$	0.01	$\times$ 1110
$\text{Att}_{\text{CoNy}}^{\text{Cont}}$ -4	2	$93.54\pm 4.91$	0.12	$\times$ 184.9
$\text{Att}_{\text{CoNy}}^{\text{Fix}}$ -4	1	$90.76\pm 5.52$	0.006	$\times$ 1850
$\text{Att}_{\text{CoNy}}^{\text{Fix}}$ -4	2	$92.36\pm 5.52$	0.10	$\times$ 221.9