notesum.ai

Published at May 10

Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

NeurIPS

Spotlight

Released Date: May 10, 2024

Authors: Wenyu Du¹, Tongxu Luo², Zihan Qiu³, Zeyu Huang⁴, Yikang Shen⁵, Reynold Cheng¹, Yike Guo², Jie Fu²

Aff.: ¹School of Computing and Data Science, The University of Hong Kong; ²HKUST; ³Tsinghua University; ⁴University of Edinburgh; ⁵MIT-IBM Watson AI Lab

Arxiv: https://openreview.net/pdf/2c7146fd80b9afb7935702797fb5bfb715dfb090.pdf