notesum.ai

Published at December 10

cs.CL

cs.AI

Released Date: December 10, 2024

Authors: Kinshuk Vasisht¹, Navreet Kaur², Danish Pruthi¹

Aff.: ¹Indian Institute of Science, Bengaluru, India; ²University of Washington, Seattle, USA

Technique	Gemma-2		LLaMA	GPT		Mistral
	2B	9B	3.1 8B	3.5-T	4o	7B
Prompt (ZS)	25.2 ( $\pm$ 3.0)	90.1 ( $\pm$ 1.9)	97.4 ( $\pm$ 0.6)	37.4 ( $\pm$ 2.9)	97.9 ( $\pm$ 0.7)	97.4 ( $\pm$ 0.7)
Prompt (FS+CoT)	99.4 ( $\pm$ 0.2)	96.3 ( $\pm$ 0.9)	97.0 ( $\pm$ 0.4)	95.6 ( $\pm$ 0.9)	99.7 ( $\pm$ 0.1)	76.1 ( $\pm$ 2.5)
Act. Steering	81.6 ( $\pm$ 1.1)	79.7 ( $\pm$ 1.4)	97.6 ( $\pm$ 0.4)	—	—	89.7 ( $\pm$ 0.7)
SFT	86.4 ( $\pm$ 1.4)	90.4 ( $\pm$ 0.9)	82.2 ( $\pm$ 1.5)	—	—	87.9 ( $\pm$ 1.4)
SFT + DPO	74.0 ( $\pm$ 2.5)	49.3 ( $\pm$ 2.8)	70.2 ( $\pm$ 2.2)	—	—	58.7 ( $\pm$ 3.5)