notesum.ai

Published at October 23

PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context

cs.AI

cs.CL

cs.MA

Released Date: October 23, 2024

Authors: Maximilian Augustin¹, Syed Shakib Sarwar², Mostafa Elhoushi³, Sai Qian Zhang², Yuecheng Li², Barbara De Salvo²

Aff.: ¹Tübingen AI Center University of Tübingen; ²Reality Labs Research at Meta; ³FAIR at Meta

Arxiv: https://arxiv.org/abs/2410.17661v1

Refer to caption

Model	Sparsity	IN21K acc.	#params	Gflops	NPU Latency (ms)
EfficientFormer L7	-	50.43	80M	20.3	6.9
EfficientFormer L3	-	47.72	31M	7.84	3.0
ViT-B	-	49.99	85M	35.13	18.2
ViT-S	-	46.02	22M	9.20	9.0
EfficientFormer L7	0.9	47.11	7.9M	20.3	6.9
ViT-B	0.9	46.98	8.1M	35.13	18.2