notesum.ai

Published at November 26

An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models

cs.LG

Released Date: November 26, 2024

Authors: Yunzhe Hu¹, Difan Zou², Dong Xu¹

Aff.: ¹School of Computing and Data Science, The University of Hong Kong; ²School of Computing and Data Science & Institute of Data Science, The University of Hong Kong

Arxiv: http://arxiv.org/abs/2411.17182v1

Models	CIFAR-10		CIFAR-100
Models	cross-entropy	+ SRR regularization (L=12)	cross-entropy	+ SRR regularization (L=12)
CRATE-C	76.87	77.61	43.40	44.53
CRATE-N	81.52	81.91	55.11	55.62
CRATE-T	85.49	85.52	60.59	60.69
CRATE	86.67	86.79	62.40	62.52