notesum.ai

Published at October 30

Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes

stat.ML

cs.AI

cs.LG

cs.NE

Released Date: October 30, 2024

Authors: Jerry Yao-Chieh Hu¹, Dennis Wu¹, Han Liu¹

Aff.: ¹Northwestern University

Arxiv: http://arxiv.org/abs/2410.23126v1

Refer to caption

	Synthetic						MNIST
	Softmax		$1.5$ -entmax		sparsemax		Softmax		$1.5$ -entmax		sparsemax
$\norm{p}_{0}$	-	$\mathtt{U}\text{-}\mathtt{Hop}$ +	-	$\mathtt{U}\text{-}\mathtt{Hop}$ +	-	$\mathtt{U}\text{-}\mathtt{Hop}$ +	-	$\mathtt{U}\text{-}\mathtt{Hop}$ +	-	$\mathtt{U}\text{-}\mathtt{Hop}$ +	-	$\mathtt{U}\text{-}\mathtt{Hop}$ +
1	$0.0$	$90.0$	$0.0$	$100.0$	$0.0$	$100.0$	$3.48$	$100.0$	$69.2$	$100.0$	$88.1$	$100.0$
2	$0.0$	$8.0$	$0.0$	$0.0$	$20.0$	$0.0$	$2.16$	$0.0$	$8.6$	$0.0$	$5.2$	$0.0$
3	$0.0$	$0.0$	$0.0$	$0.0$	$30.0$	$0.0$	$1.57$	$0.0$	$3.9$	$0.0$	$2.6$	$0.0$
4	$0.0$	$2.0$	$0.0$	$0.0$	$50.0$	$0.0$	$1.23$	$0.0$	$2.3$	$0.0$	$1.6$	$0.0$
5	$6.0$	$0.0$	$0.0$	$0.0$	$0.0$	$0.0$	$1.2$	$0.0$	$1.6$	$0.0$	$1.1$	$0.0$
6	$10.0$	$0.0$	$25.0$	$0.0$	$0.0$	$0.0$	$0.95$	$0.0$	$0.9$	$0.0$	$0.8$	$0.0$
7	$20.0$	$0.0$	$25.0$	$0.0$	$0.0$	$0.0$	$1.04$	$0.0$	$0.6$	$0.0$	$0.4$	$0.0$
8	$16.0$	$0.0$	$0.0$	$0.0$	$0.0$	$0.0$	$0.84$	$0.0$	$0.6$	$0.0$	$0.1$	$0.0$
9	$20.0$	$0.0$	$22.5$	$0.0$	$0.0$	$0.0$	$1.03$	$0.0$	$0.3$	$0.0$	$0.0$	$0.0$
10⁺	$28.0$	$0.0$	$27.5$	$0.0$	$0.0$	$0.0$	$86.5$	$0.0$	$12.0$	$0.0$	$0.1$	$0.0$