notesum.ai

Published at November 19

HNCSE: Advancing Sentence Embeddings via Hybrid Contrastive Learning with Hard Negatives

cs.CL

cs.AI

Released Date: November 19, 2024

Authors: Wenxiao Liu¹, Zihong Yang¹, Chaozhuo Li², Zijin Hong³, Jianfeng Ma¹, Zhiquan Liu¹, Litian Zhang⁴, Feiran Huang¹

Aff.: ¹College of Cyberspace Security, Jinan University; ²School of Cyberspace Security, Beijing University of Posts and Telecommunications; ³Birmingham College, Jinan University; ⁴Beihang University

Arxiv: http://arxiv.org/abs/2411.12156v1

Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK-R	Avg.
Unsupervised Models (Base)
GloVe (avg.)	55.14	70.66	59.73	68.25	63.66	58.02	53.76	61.32
BERT (first-last avg.)	39.70	59.38	49.67	66.03	66.19	53.87	62.06	56.70
BERT-flow	58.40	67.10	60.85	75.16	71.22	68.66	64.47	66.55
BERT-whitening	57.83	66.90	60.90	75.08	71.31	68.24	63.73	66.28
IS-BERT	56.77	69.24	61.21	75.23	70.16	69.21	64.25	66.58
CT-BERT	61.63	76.80	68.47	77.50	76.48	74.31	69.19	72.05
RoBERTa (first-last avg.)	40.88	58.74	49.07	65.63	61.48	58.55	61.63	56.57
RoBERTa-whitening	46.99	63.24	57.23	71.36	68.99	61.36	62.91	61.73
DeCLUTR-RoBERT	52.41	75.19	65.52	77.12	78.63	72.41	68.62	69.99
SIMCSE	68.40	82.41	74.38	80.91	78.56	76.85	72.23	76.25
SIMCSE_(reproduce)	70.82	82.24	73.25	81.38	77.06	77.24	71.16	76.16
LLaMA2-7B	50.66	73.32	62.76	67.00	70.98	63.28	67.40	65.06
LLaMA2-7B_(PromptEOL)	58.81	77.01	66.34	73.22	73.56	71.66	69.64	70.03
LLaMA2-7B_{(Pretended_CoT)}	67.45	83.89	74.14	79.47	80.76	78.95	73.33	76.86
LLaMA2-7B_{(Konwledge_Enhancement)}	65.60	82.82	74.48	80.75	80.13	80.34	75.89	77.14
HNCSE-PM(ours)	71.02	83.92	75.52	82.93	81.03	81.45	72.76	78.38
HNCSE-HNM(ours)	69.76	83.97	75.52	83.21	81.63	81.85	72.87	78.27
Unsupervised Models (Large)
SIMCSE	70.88	84.16	76.43	84.50	79.76	79.26	73.88	78.11
SIMCSE_(reproduce)	71.02	83.52	76.06	83.83	78.95	79.26	72.24	77.84
HNCSE-PM(ours)	72.94	84.67	77.24	83.97	79.53	80.78	74.79	79.13
HNCSE-HNM(ours)	72.75	84.54	77.36	84.58	79.92	80.60	74.64	79.20