notesum.ai

Published at November 28

cs.SE

cs.AI

Released Date: November 28, 2024

Authors: Amir Tavanaei¹, Kee Kiat Koo¹, Hayreddin Ceker¹, Shaobai Jiang¹, Qi Li¹, Julien Han¹, Karim Bouyarmane¹

Aff.: ¹Amazon, Seattle, USA

Model	Relative Cost	Precision	Recall	TQ	FBQ
Input dataset (baseline)	–	85.31	46.69	42.24	55.39
SoLM Self-Supervised	1X (1 run)	83.30	60.20	57.93	98.62
SoLM SFT	1X (1 run)	82.30	65.70	72.99	98.62
Claude 3.0 Sonnet (single prompt)	6.8X (1 run)	83.90	67.40	66.47	99.67
Mixtral-8x7B-Instruct (1 run per attribute)	2X (M runs)	81.80	58.4	58.37	76.62