notesum.ai

Published at October 18

GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings

cs.RO

cs.AI

cs.ET

cs.HC

cs.SY

eess.SY

Released Date: October 18, 2024

Authors: Raghuveer Thirukovalluru¹, Bhuwan Dhingra¹

Aff.: ¹Duke University

Arxiv: https://arxiv.org/abs/2410.14635v1

Method	Generator	$\mathbf{m}$	STS12	STS13	STS14	STS15	STS16	STSB	Sick-R	Avg.
Embedder: Mistral0.1-7B
Token Avg.^†	-	-	41.13	54.08	43.99	56.94	53.80	42.99	52.32	49.32
Echo^†			58.43	78.53	68.42	78.82	77.52	73.85	71.95	72.5
PromptEOL^†			63.08	78.58	69.40	77.92	79.01	75.77	69.47	73.32
PCoTEOL^†			66.45	82.04	72.24	77.93	79.36	76.66	71.06	75.11
KEEOL^†			66.33	81.52	71.73	77.53	77.99	74.09	74.02	73.89
MetaEOL^†			64.05	82.35	71.57	81.36	79.98	78.29	75.13	76.09
GenEOL	Mistral0.1-I-7B	8	69.60	83.66	77.99	82.46	81.93	81.67	76.44	79.11 (+3.02)
	Mistral0.1-I-7B	32	72.13	84.65	79.48	82.83	82.44	81.56	76.83	79.99 (+3.9)
	ChatGPT	8	70.10	83.92	78.73	82.41	81.96	80.37	77.49	79.28 (+3.19)
	ChatGPT	32	72.38	84.62	79.19	82.74	81.85	81.14	77.76	79.96 (+3.87)
Embedder: Llama2-7B
Token Avg.^†	-	-	35.49	53.15	40.12	55.35	53.26	42.10	49.96	47.06
Echo^†			52.40	72.40	61.24	72.67	73.51	65.73	64.39	66.05
PromptEOL^†			58.81	77.01	66.34	73.22	73.56	71.66	69.64	70.03
PCoTEOL^†			67.45	83.89	74.14	79.47	80.76	78.95	73.33	76.86
KEEOL^†			66.60	82.62	74.48	80.75	80.13	80.34	75.89	77.14
MetaEOL^†			64.16	81.61	73.09	81.11	79.84	77.96	74.46	75.96
GenEOL	Mistral0.1-I-7B	8	67.37	82.48	76.50	81.33	79.81	80.03	77.68	77.89 (+0.75)
	Mistral0.1-I-7B	32	70.24	83.43	78.03	81.79	80.65	80.46	78.08	78.95 (+1.81)
	ChatGPT	8	68.39	82.48	77.00	82.05	80.28	79.20	78.38	78.25 (+1.11)
	ChatGPT	32	70.78	83.28	77.75	82.10	80.45	79.83	78.71	78.99 (+1.85)
Embedder: Llama3-8B
PromptEOL^†	-	-	60.88	78.57	68.18	76.75	77.16	72.83	68.94	71.90
PCoTEOL			65.38	82.44	71.26	79.22	79.80	77.99	72.49	75.51
KEEOL			62.18	82.35	73.04	80.13	80.17	78.95	77.23	76.29
MetaEOL^†			65.10	83.08	73.01	81.57	81.47	80.47	76.46	77.35
GenEOL	Mistral0.1-I-7B	8	68.71	84.24	77.95	82.13	82.44	81.80	78.74	79.43 (+2.08)
	Mistral0.1-I-7B	32	71.39	85.13	79.63	82.44	82.79	82.16	79.03	80.37 (+3.02)
	ChatGPT	8	68.63	84.03	78.44	81.98	82.57	80.45	79.12	79.32 (+1.97)
	ChatGPT	32	71.07	84.67	78.90	82.09	82.66	81.15	79.33	79.98 (+2.63)