notesum.ai

Published at November 4

See it, Think it, Sorted: Large Multimodal Models are Few-shot Time Series Anomaly Analyzers

cs.LG

cs.AI

stat.ML

Released Date: November 4, 2024

Authors: Jiaxin Zhuang¹, Leon Yan¹, Zhenwei Zhang¹, Ruiqi Wang¹, Jiawei Zhang¹, Yuantao Gu¹

Aff.: ¹Tsinghua University

Arxiv: http://arxiv.org/abs/2411.02465v1

Dataset	UCR						NASA-SMAP						NASA-MSL
Metric	F1%		AUC-PR%		AUC-ROC%		F1%		AUC-PR%		AUC-ROC%		F1%		AUC-PR%		AUC-ROC%
IF	24.7	77.3	37.7	44.6	24.4	25.3	54.2	94.2	58.9	77.1	65.0	87.7	47.6	88.6	53.6	80.4	68.7	88.7
LOF	42.8	100	35.6	50.0	92.8	99.9	62.2	100	43.4	61.4	60.1	99.9	36.4	66.8	44.5	66.0	58.6	99.8
TranAD	38.2	93.7	30.9	51.0	77.0	99.9	59.0	99.6	36.8	73.9	74.4	100	64.6	99.1	49.2	79.6	82.5	99.9
GDN	71.4	80.6	33.4	59.0	87.1	99.9	76.4	100	40.8	66.2	86.1	100	85.1	100	38.7	56.7	93.8	100
MAD_GAN	74.2	85.0	51.5	65.9	99.4	99.9	61.3	100	39.9	72.3	83.3	100	96.0	100	46.4	50.0	95.7	100
MSCRED	32.6	96.0	28.9	45.9	94.2	99.9	57.0	97.9	40.8	61.7	77.0	100	63.0	92.2	39.5	51.8	73.2	98.1
MTAD_GAT	14.8	36.6	34.2	38.9	84.6	94.4	78.3	100	40.2	58.0	77.0	100	90.6	100	49.2	67.8	81.2	100
OmniAnomaly	34.5	95.7	26.0	45.9	85.6	99.9	57.1	100	43.6	63.2	77.5	100	71.4	100	40.0	74.9	85.0	99.9
USAD	57.6	100	33.1	50.0	97.1	99.9	72.8	100	43.6	63.2	93.9	100	91.6	99.9	42.6	60.8	94.2	100
TimesNet	32.8	45.8	15.4	23.5	98.4	99.4	97.7	100	51.4	90.3	99.8	100	97.4	100	52.9	79.7	99.8	100
SIGLLM (GPT-4o)	23.1	44.6	7.40	15.5	93.5	96.5	69.0	97.8	29.1	49.2	95.5	99.8	70.7	97.9	72.4	100	90.0	100
TAMA	92.5	97.6	93.0	97.7	99.8	99.9	94.5	100	95.5	100	98.4	100	97.5	100	99.4	100	99.8	100
TAMA*	92.5	97.6	93.0	97.7	99.8	99.9	87.8	100	89.2	100	97.0	100	96.1	100	97.7	100	99.0	100
Dataset	SMD						ECG						Dodgers
Metric	F1%		AUC-PR%		AUC-ROC%		F1%		AUC-PR%		AUC-ROC%		F1%		AUC-PR%		AUC-ROC%
IF	83.9	100	73.8	97.0	99.5	100	80.8	99.0	73.4	92.2	97.2	100	48.4	48.4	52.2	52.2	89.4	89.4
LOF	27.8	75.2	39.9	64.6	52.9	59.3	21.8	39.8	41.4	60.7	56.3	84.2	45.3	45.3	40.8	40.8	63.0	63.0
TranAD	77.0	99.6	70.9	91.0	96.8	100	69.1	98.9	74.7	97.7	94.9	100	38.2	38.2	33.9	33.9	74.6	74.6
GDN	76.9	99.7	55.0	88.6	77.0	100	75.2	96.2	76.6	97.4	96.9	99.9	37.0	37.0	31.3	31.3	74.2	74.2
MAD_GAN	67.1	92.6	61.7	87.5	91.7	100	79.1	99.3	79.2	97.6	96.9	100	32.2	32.2	28.6	28.6	74.7	74.7
MSCRED	69.4	95.7	55.8	96.2	95.0	100	66.4	99.0	73.8	97.2	88.7	100	37.8	37.8	30.6	30.6	74.6	74.6
MTAD_GAT	69.7	95.3	59.5	90.4	90.6	99.7	67.5	100	73.5	98.8	82.5	100	39.1	39.1	36.0	36.0	74.9	74.9
OmniAnomaly	66.0	96.4	61.6	91.8	87.7	100	76.8	98.6	76.4	97.5	93.5	100	33.6	33.6	35.4	35.4	60.3	60.3
USAD	72.2	99.7	67.8	93.5	94.4	100	71.5	96.9	75.2	98.3	94.9	100	37.8	37.8	33.1	33.1	74.6	74.6
TimesNet	82.8	100	57.3	99.9	95.4	100	92.4	96.6	90.0	97.6	99.4	100	48.1	48.1	73.0	73.0	83.7	83.7
SIGLLM (GPT-4o)	42.9	59.8	30.4	53.1	68.8	77.8	19.2	50.4	71.0	87.6	94.2	96.9	48.1	48.1	60.7	60.7	83.2	83.2
TAMA	77.8	100	87.9	100	98.9	100	81.3	87.5	84.5	90.0	95.4	99.4	65.6	65.6	74.0	74.0	85.2	85.2
TAMA*	62.8	93.0	78.6	97.2	99.7	99.7	78.1	88.0	83.4	91.1	94.7	99.1	64.5	64.5	73.6	73.6	85.3	85.3