notesum.ai

Published at October 31

Length-Induced Embedding Collapse in Transformer-based Models

cs.CL

cs.AI

cs.IR

Released Date: October 31, 2024

Authors: Yuqi Zhou¹, Sunhao Dai¹, Zhanshuo Cao¹, Xiao Zhang¹, Jun Xu¹

Aff.: ¹Gaoling School of Artificial Intelligence, Renmin University of China

Arxiv: http://arxiv.org/abs/2410.24200v1

Refer to caption

	Class.	Clust.	Summ.	STS	BeirRetr.	Rerank.	LongEmbdRetr.	Avg.
Num. Datasets $(\rightarrow)$	8	11	1	10	2	4	4	40
window=512
ANCE	55.27	33.04	29.58	66.32	36.87	49.09	34.02	43.45
$+\text{ours}(\tau=0.9)$	55.37	33.28	29.56	66.47	36.86	49.25	33.93	43.53
Relative Improv. (%)	▼ 0.17 ▲	▼ 0.73 ▲	▼ -0.05 ▼	▼ 0.22 ▲	▼ -0.01 ▼	▼ 0.32 ▲	▼ -0.25 ▼	▼ 0.18 ▲
GTR	55.10	38.65	29.67	70.11	44.98	54.23	37.33	47.15
$+\text{ours}(\tau=0.8)$	55.51	39.52	29.83	70.26	45.61	54.16	37.33	47.46
Relative Improv. (%)	▼ 0.73 ▲	▼ 2.26 ▲	▼ 0.54 ▲	▼ 0.21 ▲	▼ 1.41 ▲	▼ -0.13 ▼	▼ 0.01 ▲	▼ 0.65 ▲
GIST	64.75	44.77	31.14	75.61	52.77	58.55	38.21	52.26
$+\text{ours}(\tau=0.9)$	65.00	44.64	31.17	75.59	53.41	58.60	38.35	52.39
Relative Improv. (%)	▼ 0.38 ▲	▼ -0.29 ▼	▼ 0.09 ▲	▼ -0.03 ▼	▼ 1.21 ▲	▼ 0.08 ▲	▼ 0.36 ▲	▼ 0.26 ▲
BGE	64.79	45.80	31.03	75.88	55.29	58.87	37.46	52.73
$+\text{ours}(\tau=0.8)$	64.89	45.61	31.51	75.68	56.00	58.97	38.35	53.00
Relative Improv. (%)	▼ 0.16 ▲	▼ -0.42 ▼	▼ 1.53 ▲	▼ -0.26 ▼	▼ 1.29 ▲	▼ 0.17 ▲	▼ 2.40 ▲	▼ 0.51 ▲
window=4k
E5	61.72	38.82	30.58	71.77	47.22	53.12	56.01	51.32
$+\text{ours}(\tau=0.8)$	62.15	40.22	31.11	72.17	47.06	53.47	56.88	51.87
Relative Improv. (%)	▼ 0.70 ▲	▼ 3.61 ▲	▼ 1.74 ▲	▼ 0.55 ▲	▼ -0.33 ▼	▼ 0.65 ▲	▼ 1.56 ▲	▼ 1.07 ▲
Avg Improv. (%)	▼ 0.43 ▲	▼ 1.18 ▲	▼ 0.77 ▲	▼ 0.14 ▲	▼ 0.71 ▲	▼ 0.22 ▲	▼ 0.82 ▲	▼ 0.53 ▲