notesum.ai

Published at November 12

Problem-Oriented Segmentation and Retrieval: Case Study on Tutoring Conversations

cs.CL

cs.AI

Released Date: November 12, 2024

Authors: Rose E. Wang¹, Pawan Wirawarn¹, Kenny Lam¹, Omar Khattab¹, Dorottya Demszky¹

Aff.: ¹Stanford University

Arxiv: http://arxiv.org/abs/2411.07598v1

		POSR Metrics
Segmentation	Retrieval	SRS ( $\uparrow$ )		Cost ( $\downarrow$ )
Method	Method	Line	Time
Opus	Jaccard	$0.62\pm 0.19$	$0.63\pm 0.19$	$17.16\pm 4.81$
Opus	TFIDF	$0.63\pm 0.22$	$0.63\pm 0.22$	$17.16\pm 4.81$
Opus	BM25	$0.52\pm 0.23$	$0.53\pm 0.23$	$17.16\pm 4.81$
Opus	ColBERT	$0.50\pm 0.23$	$0.50\pm 0.23$	$17.16\pm 4.81$
Opus	GPT4	$0.87\pm 0.13$	$0.88\pm 0.13$	$54.32\pm 14.71$
Opus	Haiku	$0.58\pm 0.23$	$0.59\pm 0.24$	$18.17\pm 4.90$
Opus	Sonnet	$0.70\pm 0.20$	$0.71\pm 0.20$	$29.25\pm 7.04$
Opus	Opus	$0.85\pm 0.11$	$0.86\pm 0.11$	$77.58\pm 22.54$
POSR GPT4		$\bf 0.88\pm 0.12$	$\bf 0.89\pm 0.11$	$11.72\pm 2.71$
POSR Haiku		$0.61\pm 0.22$	$0.62\pm 0.23$	$\bf 0.35\pm 0.08$
POSR Sonnet		$0.85\pm 0.14$	$0.85\pm 0.14$	$4.22\pm 0.93$
POSR Opus		$\bf 0.88\pm 0.11$	$\bf 0.89\pm 0.11$	$21.11\pm 4.64$