notesum.ai

Published at December 5

Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs

cs.AR

Released Date: December 5, 2024

Authors: Severin Bochem¹, Victor J. B. Jung, Arpan Prasad², Francesco Conti³, Luca Benini⁴

Aff.: ¹D-ITET, ETH Zurich, Switzerland; ²Integrated Systems Laboratory, ETH Zurich, Switzerland; ³DEI, and Information Engineering, University of Bologna, Italy; ⁴Integrated Systems Laboratory, ETH Zurich, Switzerland; DEI, and Information Engineering, University of Bologna, Italy

Arxiv: http://arxiv.org/pdf/2412.04372v1

Work	Model	Scale	Platform	Pipelining	Weight Duplication
Deepthings [20]	CNN	Low-Power	Raspberry Pi	No	Yes
Efficiently Scaling Transformer Inference [13]	Transformer	Datacenter	TPU	No	No
DeepSpeed Inference [12]	Transformer	Datacenter	GPU	Yes	No
When the Edge Meets Transformers [21]	Transformer	Low-Power	CPU	No	Yes
Hermes [22]	Transformer	Low-Power	CPU	Yes	No
Ours	Transformer	Extreme Edge	Siracusa (MCU)	No	No