notesum.ai

Published at December 9

Measuring Pre-training Data Quality without Labels for Time Series Foundation Models

cs.LG

cs.AI

stat.ML

Released Date: December 9, 2024

Authors: Songkang Wen¹, Vasilii Feofanov², Jianfeng Zhang¹

Aff.: ¹Huawei Noah's Ark Lab, Shenzhen, China; ²Huawei Noah's Ark Lab, Paris, France

Arxiv: http://arxiv.org/pdf/2412.06368v1

Refer to caption

Part	Parameters	Value
RandomCropResize	Crop Scale	0.7 0.8
RandomCropResize	Resize Length	512
Non-Overlapping Patch	Patch Length	16
Overlapping Patch	CNN Kernel Size	17
	CNN Padding	8
	Output Channel	256
ViT	Tokens Number	32
	CLS Tokens	1
	Token dimension	256
	Layer	6
	Number Of Head	8
	Dimension of Head	128
	MLP dimension	512
Linear for mu	layer	1
Linear for mu	dim	1x32
Linear for std	layer	1
Linear for std	dim	1x32
InfoNCE Loss	Temperature	0.1
Non-Linear Projector	LayerNorm
	Layer	2
	Input Dim	256
	Hidden Dim	512
	Output Dim	256
	Activation Function	ReLU
Classifier Linear Head	LayerNorm
	layer	1
	Dim	256xClassNum
Training Setting	Learning Rate	2e-04
	Epochs	500
	Batch Size	64
Testing Setting	Learning Rate	2e-04
	Epochs	500
	Batch Size	256
Optimizer	Type	AdamW
	Betas	0.9 0.999
	Weight Decay	0.05