notesum.ai

Published at November 14

StreamAdapter: Efficient Test Time Adaptation from Contextual Streams

cs.CL

cs.AI

Released Date: November 14, 2024

Authors: Dilxat Muhtar¹, Yelong Shen², Yaming Yang², Xiaodong Liu², Yadong Lu², Jianfeng Liu², Yuefeng Zhan², Hao Sun², Weiwei Deng², Feng Sun², Xueliang Zhang¹, Jianfeng Gao², Weizhu Chen², Qi Zhang²

Aff.: ¹Nanjing University; ²Microsoft

Arxiv: http://arxiv.org/abs/2411.09289v1

		Seen Task						Unseen Task
		BoolQ	CoPA	SST2	CB	RTE	Avg.	Hellaswag	Winogrande	OBQA	ARC-C	ARC-E	PIQA	Avg.
Zero-shot	Tiny-Llama-1.1B	57.03	78.00	69.61	14.29	51.99	54.18	59.01	58.88	21.80	27.56	60.31	73.34	50.15
ICL ${}_{10-\text{shot}}$		63.39	76.00	78.21	51.79	49.10	63.70	59.46	59.83	26.20	30.61	64.81	73.78	52.45
LoRA		74.28	78.00	86.58	80.36	69.31	77.71	59.01	56.35	24.80	27.90	57.15	72.14	49.56
TempLoRA		57.61	74.00	71.33	39.29	56.68	59.78	59.42	60.46	21.60	27.82	62.12	72.09	50.59
H₂O		62.72	77.00	75.80	23.21	47.23	57.19	58.86	59.12	23.20	30.20	62.58	72.96	51.15
SnapKV		63.29	75.00	63.30	44.64	46.93	58.63	58.09	59.83	23.40	28.75	62.92	72.80	50.97
StreamAdapter		81.77	76.00	89.56	85.71	82.67	83.14	59.45	59.91	27.00	31.06	64.93	73.29	52.61
Zero-shot	LLaMA-3-8B	81.11	88.00	67.09	51.79	67.87	71.17	79.15	72.93	35.60	50.09	80.35	79.60	66.29
ICL ${}_{10-\text{shot}}$		83.49	89.00	95.07	83.93	79.06	86.11	81.46	78.45	36.40	52.43	82.79	80.58	68.69
LoRA		89.24	84.00	96.22	96.43	88.09	90.80	80.78	69.14	35.40	51.62	80.77	78.67	66.06
TempLoRA		81.77	91.00	90.60	76.79	70.40	82.11	79.62	76.56	34.20	50.85	81.06	79.27	66.93
H₂O		82.48	88.00	92.43	73.21	75.81	82.39	80.57	75.77	33.80	51.45	82.62	79.98	67.37
SnapKV		84.46	88.00	94.41	75.00	74.01	83.18	80.41	76.95	35.20	50.77	82.74	79.65	67.62
StreamAdapter		90.15	89.00	96.72	98.21	89.67	92.75	80.87	78.64	38.80	53.03	83.53	80.59	69.24
Zero-shot	Phi-3-Medium	88.65	92.00	90.48	73.21	80.51	84.97	79.12	76.48	37.80	54.95	80.85	80.90	68.35
ICL ${}_{10-\text{shot}}$		88.23	94.00	95.07	83.93	77.62	87.77	80.70	79.40	44.80	62.80	86.87	82.37	72.82
LoRA		88.59	92.00	96.22	92.86	89.53	91.84	79.36	76.87	41.80	58.60	84.22	81.18	70.34
TempLoRA		89.27	92.00	93.81	71.43	78.34	84.97	79.30	76.56	44.40	60.58	86.28	80.61	71.29
H₂O		83.46	94.00	94.95	80.46	79.78	86.53	80.61	76.11	40.80	62.12	86.41	82.07	71.35
SnapKV		78.99	89.00	70.41	80.36	68.95	77.54	80.21	72.53	41.30	61.71	83.89	79.87	69.92
StreamAdapter		90.24	92.00	95.18	92.86	89.89	92.03	82.03	77.11	44.80	64.33	86.91	82.47	72.94