notesum.ai

Published at November 16

cs.AI

Released Date: November 16, 2024

Authors: Huaqin Zhao¹, Jiaxi Li¹, Yi Pan¹, Shizhe Liang¹, Xiaofeng Yang², Wei Liu³, Xiang Li⁴, Fei Dou¹, Tianming Liu¹, Jin Lu¹

Aff.: ¹University of Georgia; ²Emory University; ³Mayo Clinic; ⁴Massachusetts General Hospital and Harvard Medical School

Task Type	SST-2	SST-5	SNLI	MNLI	RTE	TREC
	—— sentiment ——		—— natural language inference ——			— topic —
Zero-shot	79.0	35.5	50.2	48.8	51.4	32.0
LP	76.0 ( $\pm$ 2.8)	40.3 ( $\pm$ 1.9)	66.0 ( $\pm$ 2.7)	56.5 ( $\pm$ 2.5)	59.4 ( $\pm$ 5.3)	51.3 ( $\pm$ 5.5)
FT	91.9 ( $\pm$ 1.8)	46.7 ( $\pm$ 1.9)	77.5 ( $\pm$ 2.6)	70.0 ( $\pm$ 2.3)	66.4 ( $\pm$ 7.2)	85.0 ( $\pm$ 2.5)
FT(LoRA)	91.4 ( $\pm$ 1.7)	46.7 ( $\pm$ 1.1)	74.9 ( $\pm$ 4.3)	67.7 ( $\pm$ 1.4)	66.1 ( $\pm$ 3.5)	82.7 ( $\pm$ 4.1)
FT(Prefix)	91.9 ( $\pm$ 1.0)	47.7 ( $\pm$ 1.1)	77.2 ( $\pm$ 1.3)	66.5 ( $\pm$ 2.5)	66.6 ( $\pm$ 2.0)	85.7 ( $\pm$ 1.3)
MeZO	90.5 ( $\pm$ 1.2)	45.5 ( $\pm$ 2.0)	68.5 ( $\pm$ 3.9)	58.7 ( $\pm$ 2.5)	64.0 ( $\pm$ 3.3)	76.9 ( $\pm$ 2.7)
MeZO (LoRA)	91.4 ( $\pm$ 0.9)	43.0 ( $\pm$ 1.6)	69.7 ( $\pm$ 6.0)	64.0 ( $\pm$ 2.5)	64.9 ( $\pm$ 3.6)	73.1 ( $\pm$ 6.5)
MeZO (Prefix)	90.8 ( $\pm$ 1.7)	45.8 ( $\pm$ 2.0)	71.6 ( $\pm$ 2.5)	63.4 ( $\pm$ 1.8)	65.4 ( $\pm$ 3.9)	80.3 ( $\pm$ 3.6)
HELENE	91.6 ( $\pm$ 2.3)	44.7 ( $\pm$ 0.8)	70.0 ( $\pm$ 2.6)	58.9 ( $\pm$ 1.1)	65.7 ( $\pm$ 1.2)	78.1 ( $\pm$ 1.5)
HELENE (LoRA)	90.6 ( $\pm$ 0.3)	41.8 ( $\pm$ 1.0)	68.5 ( $\pm$ 2.0)	59.0 ( $\pm$ 1.1)	66.8 ( $\pm$ 3.2)	67.4 ( $\pm$ 2.1)
HELENE (Prefix)	91.7 ( $\pm$ 0.6)	46.0 ( $\pm$ 0.7)	69.5 ( $\pm$ 1.9)	64.6 ( $\pm$ 2.1)	66.1 ( $\pm$ 1.8)	77.4 ( $\pm$ 2.1)