notesum.ai

Published at October 21

cs.AI

Released Date: October 21, 2024

Authors: Tian Yu Liu¹, Stefano Soatto¹

Aff.: ¹Department of Computer Science, University of California, Los Angeles

	STS-B	STS12	STS13	STS14	STS15	STS16	SICK-R	Avg
Contrastive-Trained Embedding Models
CLIP-ViTL14^‡ (Radford et al., 2021)	65.5	67.7	68.5	58.0	67.1	73.6	68.6	67.0 $\pm$ 4.3
IS-BERT^† (Zhang et al., 2020)	56.8	69.2	61.2	75.2	70.2	69.2	64.3	66.6 $\pm$ 5.7
SimCSE-BERT^† (Gao et al., 2021)	68.4	82.4	74.4	80.9	78.6	76.9	72.2	76.3 $\pm$ 4.6
Zero-Shot Encoder-based Models
BERT-CLS^∗ (Devlin et al., 2018)	16.5	20.2	30.0	20.1	36.9	38.1	42.6	29.2 $\pm$ 9.6
BERT-mean^∗ (Devlin et al., 2018)	45.4	38.8	58.0	58.0	63.1	61.1	58.4	54.8 $\pm$ 8.3
BERT Large-mean^∗ (Devlin et al., 2018)	47.0	27.7	55.8	44.5	51.7	61.9	53.9	48.9 $\pm$ 10.2
RoBERTa Large-mean^∗ (Liu et al., 2019)	50.6	33.6	57.2	45.7	63.0	61.2	58.4	52.8 $\pm$ 9.6
ST5-Enc-mean (Large)^∗ (Ni et al., 2021)	56.3	28.0	52.6	41.4	61.3	63.6	59.5	51.8 $\pm$ 11.9
ST5-Enc-mean (11B)^∗ (Ni et al., 2021)	62.8	35.0	60.2	47.6	66.4	70.6	63.6	58.0 $\pm$ 11.5
Autoregressive Models (Meanings as Trajectories (Liu et al., 2023))
GPT-2^‡	55.2	39.9	42.6	30.5	52.4	62.7	62.0	49.3 $\pm$ 11.2
GPT-2-XL^‡	62.1	43.6	54.8	37.7	61.3	68.2	68.4	56.5 $\pm$ 11.1
Falcon-7B^‡	67.7	56.3	66.5	53.0	67.4	75.5	73.5	65.7 $\pm$ 7.7
LLaMA-13B^‡	70.6	52.5	65.9	53.2	67.8	74.1	73.0	65.3 $\pm$ 8.3
LLaMA-33B^‡	71.5	52.5	70.6	54.6	69.1	75.2	73.0	66.6 $\pm$ 8.5
Text-Conditioned Diffusion Models (Ours)
Stable Diffusion	70.3	57.9	61.0	60.8	73.6	67.9	66.0	65.4 $\pm$ 5.3