notesum.ai

Published at October 23

MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers

cs.CV

cs.AI

Released Date: October 23, 2024

Authors: Zebin Yang¹, Renze Chen², Taiqiang Wu³, Ngai Wong³, Yun Liang⁴, Runsheng Wang⁵, Ru Huang⁵, Meng Li⁴

Aff.: ¹Institute for Artificial Intelligence, Peking University, Beijing, China; ²School of Computer Science, Peking University, Beijing, China; ³The University of Hong Kong, Hong Kong, China; ⁴School of Integrated Circuits, Peking University, Beijing, China; ⁵Beijing Advanced Innovation Center for Integrated Circuits, Beijing, China

Arxiv: https://arxiv.org/abs/2410.17957v1

Notations	Meanings
$v$	Vocabulary size
$s,h,d$	Sequence length, # heads, and embedding dimension
$c$	# clusters
$i,j,l$	Loop variables for clusters, tokens, and singular values
$t$	Sequence length each tile
$U,V$	Unitary matrices generated by SVD
$U_{i},V_{i}$	Embedding table and linear projection for the $i_{th}$ cluster
$U_{i,j}$	Embedding vector for the $j_{th}$ token in $i_{th}$ cluster
$\Sigma$	Vector of singular values
$\alpha,\beta$	NAS parameters for embedding compression
$\alpha_{j,i}$	NAS parameters for $j_{th}$ token in $i_{th}$ cluster in first stage NAS
$\beta_{i,l}$	NAS parameters for $l_{th}$ singular value in $i_{th}$ cluster in second stage NAS
$\beta_{i}^{*}$	Threshold for NAS parameters in $i_{th}$ cluster in second stage NAS
$M,N,K$	Loop ranges of a matrix multiplication
$m,n,k$	Loop variables of a matrix multiplication