notesum.ai

Published at October 18

cs.CL

cs.AI

stat.ML

Released Date: October 18, 2024

Authors: Juyeon Heo¹, Christina Heinze-Deml², Oussama Elachqar³, Shirley Ren², Udhay Nallasamy², Andy Miller², Kwan Ho Ryan Chan⁴, Jaya Narain²

Aff.: ¹University of Cambridge; ²Apple; ³Not specified; ⁴UPenn

Model	Original SR	Random SR	Inst-follow SR	Original QR	Random QR	Inst-follow QR
LLaMA-2-chat-7B	0.57 $\pm$ 0.00	0.55 $\pm$ 0.00	0.59 $\pm$ 0.00	0.87 $\pm$ 0.09	0.85 $\pm$ 0.10	0.87 $\pm$ 0.08
LLaMA-2-chat-13B	0.61 $\pm$ 0.00	0.54 $\pm$ 0.12	0.65 $\pm$ 0.02	0.92 $\pm$ 0.00	0.91 $\pm$ 0.02	0.94 $\pm$ 0.00
Mistral-7B-inst-v0.3	0.58 $\pm$ 0.00	0.56 $\pm$ 0.02	0.64 $\pm$ 0.02	0.95 $\pm$ 0.02	0.86 $\pm$ 0.02	0.98 $\pm$ 0.06
Phi-3-mini-128k	0.71 $\pm$ 0.00	0.63 $\pm$ 0.04	0.74 $\pm$ 0.01	0.76 $\pm$ 0.01	0.76 $\pm$ 0.01	0.78 $\pm$ 0.00