notesum.ai

Published at November 3

Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envision

cs.LG

cs.AI

Released Date: November 3, 2024

Authors: Xiangzhong Luo¹, Di Liu², Hao Kong¹, Shuo Huai¹, Hui Chen¹, Guochu Xiong¹, Weichen Liu¹

Aff.: ¹Nanyang Technological University, Singapore; ²Norwegian University of Science and Technology, Norway

Arxiv: http://arxiv.org/abs/2411.01431v1

Benchmark	Search Space		Queryable		Tasks	Datasets	Metrics
Benchmark	Size	Type	Tabular	Surrogate	Tasks	Datasets	Metrics
NAS-Bench-101 (ying2019nasbench101, )	423k	Cell-Based	✓	✗	Image Classification	CIFAR-10	Training Accuracy, Validation Accuracy, Testing Accuracy, Training Time, and Number of Parameters
NAS-Bench-201 (dong2020nasbench201, )	15.6k	Cell-Based	✓	✗	Image Classification	CIFAR-10, CIFAR-100, and ImageNet-16-120	Training Accuracy, Validation Accuracy, Testing Accuracy, Training Loss, Validation Loss, Testing Loss, Training Time, Number of FLOPs, and Number of Parameters
NATS-Bench (dong2021nats-bench, )	39.3k	Cell-Based	✓	✗	Image Classification	CIFAR-10, CIFAR-100, and ImageNet-16-120	Training Accuracy, Validation Accuracy, Testing Accuracy, Training Loss, Validation Loss, Testing Loss, Training Time, Number of FLOPs, and Number of Parameters
NAS-Bench-301 (siems2020nasbench301, )	$10^{18}$	Cell-Based	✗	✓	Image Classification	CIFAR-10	Validation Accuracy
NAS-Bench-360 (tu2022nasbench360, )	N/A	Cell- and Block-Based	✓	✗	10 Diverse Tasks	10 Diverse Datasets	N/A
NAS-Bench-1Shot1 (zela2020nasbench1shot1, )	399k	Cell-Based	✓	✗	Image Classification	CIFAR-10	Validation Accuracy
NAS-Bench-ASR (mehrotra2021nasbenchasr, )	8.2k	Cell-Based	✓	✗	Automatic Speech Recognition	TIMIT	CTC Loss, Phoneme Error Rate (PER), On-Device Latency, Number of FLOPs, and Number of Parameters
NAS-Bench-Graph (qin2022nasbenchgraph, )	26.2k	Cell-Based	✓	✗	9 Graph Tasks	9 Graph Datasets	Training Loss, Validation Loss, Testing Loss, Validation Accuracy, On-Device Latency, and Number of Parameters
NAS-Bench-NLP (klyuchnikov2022nasbenchnlp, )	14k	Cell-Based	✓	✗	Language Understanding	PTB and WikiText-2	Testing Perplexity, Training Time, and Number of Parameters
NAS-Bench-111 (yan2021nasbenchx11, )	423k	Cell-Based	✗	✓	Image Classification	CIFAR-10	Training Accuracy, Validation Accuracy, Testing Accuracy, Training Loss, Validation Loss, and Testing Loss
NAS-Bench-311 (yan2021nasbenchx11, )	$10^{18}$	Cell-Based	✗	✓	Image Classification	CIFAR-10	Same as NAS-Bench-111
NAS-Bench-NLP11 (yan2021nasbenchx11, )	$10^{53}$	Cell-Based	✗	✓	Language Understanding	PTB	Same as NAS-Bench-111
NAS-Bench-Suite (mehta2022nasbenchsuite, )	N/A	Cell-Based	✓	✓	A suite of 11 tabular and surrogate NAS benchmarks
HW-NAS-Bench (li2021hwnasbench, )	15.6k	Cell-Based	✓	✗	Image Classification	CIFAR-10, CIFAR-100, and ImageNet-16-120	On-Device Latency
HW-NAS-Bench (li2021hwnasbench, )	$10^{21}$	Block-Based	✗	✓	Image Classification	CIFAR-100 and ImageNet	On-Device Latency