notesum.ai

Published at November 5

Navigating Extremes: Dynamic Sparsity in Large Output Space

cs.LG

cs.AI

Released Date: November 5, 2024

Authors: Nasib Ullah¹, Erik Schultheis¹, Mike Lasby², Yani Ioannou², Rohit Babbar³

Aff.: ¹Department of Computer Science, Aalto University, Helsinki, Finland; ²Schulich School of Engineering, University of Calgary, Calgary, AB, Canada; ³Department of Computer Science, Aalto University, Helsinki, Finland; Department of Computer Science, University of Bath, Bath, UK

Arxiv: http://arxiv.org/abs/2411.03171v2

Method	Sparsity (%)	P@1	P@3	P@5	$M_{\mathrm{tr}}$ $(\mathrm{GiB})$	Sparsity (%)	P@1	P@3	P@5	$M_{\mathrm{tr}}$ $(GiB)$
		Wiki10-31K					Wiki-500K
AttentionXML	-	87.1	77.8	68.8	8.2	-	75.1	56.5	44.4	13.1
LightXML	-	87.8	77.3	68.0	16.5	-	76.2	57.2	44.1	14.6
CascadeXML	-	88.4	78.3	68.9	8.2	-	77.0	58.3	45.1	18.8
Dense	-	87.8	77.2	68.1	2.5	-	78.5	59.2	45.6	9
Dense BN	-	86.7	76.3	66.0	2.1	-	73.8	55.1	42.0	4.3
RigL	92	87.7	77.3	67.7	2.6	83	74.5	54.7	41.8	9.7
Spartex	92	88.6	77.7	67.4	2.1	83	76.7	57.8	44.5	4.1
		Amazon-670K					Amazon-3M
AttentionXML	-	45.7	40.7	36.9	10.7	-	49.1	46.0	43.9	71.2
LightXML	-	47.1	42.0	38.2	11.2	-	-	-		OOM
CascadeXML	-	48.5	43.7	40.0	18.3	-	51.3	49.0	46.9	87.0
Dense	-	49.8	44.2	40.1	11.5	-	53.4	50.6	48.5	46.3
Dense BN	-	44.5	39.7	36.1	4.0	-	47.0	44.6	42.7	13.1
RigL	83	45.2	38.7	36.0	12.4	83	-	-	-	OOM
Spartex	83	47.1	41.8	38.0	3.7	83	50.2	47.1	44.8	13.5