notesum.ai

Published at December 10

Bilingual BSARD: Extending Statutory Article Retrieval to Dutch

cs.CL

cs.IR

Released Date: December 10, 2024

Authors: Ehsan Lotfi¹, Nikolay Banar¹, Nerses Yuzbashyan¹, Walter Daelemans¹

Aff.: ¹University of Antwerp

Arxiv: http://arxiv.org/pdf/2412.07462v1

Refer to caption

T	Model	Size	R@100	R@200	R@500	MAP@100	MRR@100	nDCG@10	nDCG@100
	TF-IDF	-	39.21	46.38	52.76	8.53	14.25	12.38	16.74
	BM25	-	40.19	47.95	54.57	16.07	22.63	20.07	23.57
	word2vec	-	41.06	51.02	58.94	8.28	15.27	11.66	17.05
	fastText	-	31.47	38.26	49.08	7.27	12.67	8.89	13.86
	mE5_small	118M	45.43	52.25	61.10	13.42	21.79	17.67	22.79
	mContriever	178M	47.92	58.38	68.32	11.38	20.15	14.83	21.82
	DPR-XM	277M	40.44	46.12	53.16	13.57	21.78	16.40	21.79
	mE5_base	278M	50.14	57.68	65.30	16.47	25.64	20.81	26.49
	mGTE	305M	52.78	61.97	73.09	15.86	24.92	20.08	26.80
	LaBSE	471M	20.51	28.42	42.18	2.34	6.60	3.50	7.18
	mE5_large	560M	58.35	65.83	70.83	21.88	34.28	28.47	33.51
	mE5_{large-instruct}	560M	59.48	66.80	75.21	18.66	29.93	24.84	31.33
	BGE-M3	568M	61.12	67.20	77.56	18.31	30.40	24.04	31.21
	jina-embeddings-v3	572M	60.70	67.92	77.37	18.59	31.21	24.70	31.58
	E5_mistral-7b	7B	68.35	73.91	82.82	30.24	43.26	37.70	43.02
	BGE-Mult.-Gemma2	9B	69.94	76.23	81.28	25.07	37.66	30.95	39.11
	voyage-3	-	73.08	79.37	85.67	32.81	46.38	40.06	46.21
	embedding-3-large	-	75.70	80.22	88.24	29.73	42.99	36.83	44.40
✓	Tik-to-Tok_base	116M	73.90	79.02	83.29	39.24	45.69	42.75	49.90
✓	RobBERT-2023_base	125M	75.08	79.33	83.40	40.51	47.68	44.76	51.36
✓	XLM-Roberta_base	279M	62.06	68.26	75.40	26.61	32.00	30.65	37.10