notesum.ai

Published at May 8

A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention

NeurIPS

Spotlight

Released Date: May 8, 2024

Authors: Hugo Cui¹, Freya Behrens¹, Florent Krzakala², Lenka Zdeborova

Aff.: ¹Statistical Physics of Computation laboratory EPFL, Lausanne, Switzerland; ²Information, Learning and Physics laboratory EPFL, Lausanne, Switzerland

Arxiv: https://openreview.net/pdf/411f5d2e925c9100cd435ceb6cd2b1badcbe1f52.pdf