notesum.ai

Published at November 6

Lexicalization Is All You Need: Examining the Impact of Lexical Knowledge in a Compositional QALD System

cs.AI

cs.CL

cs.IR

Released Date: November 6, 2024

Authors: David Maria Schmidt¹, Mohammad Fazleh Elahi¹, Philipp Cimiano¹

Aff.: ¹Bielefeld University

Arxiv: http://arxiv.org/abs/2411.03906v1

Refer to caption

			Without Lexicon						With Lexicon
			Micro			Macro			Micro			Macro
Model	FT	Pr#	$F_{1}$	P	R	$F_{1}$	P	R	$F_{1}$	P	R	$F_{1}$	P	R
GPT-3.5-Turbo	✗	1	$\bm{0.15}$	$0.19$	$\bm{0.12}$	$0.10$	$0.11$	$0.12$	$0.13$	$0.09$	$\bm{0.21}$	$0.25$	$\bm{0.27}$	$0.27$
GPT-3.5-Turbo	✗	2	$\bm{0.15}$	$\bm{0.21}$	$\bm{0.12}$	$0.11$	$0.10$	$0.12$	$0.13$	$0.09$	$0.20$	$\bm{0.26}$	$0.26$	$\bm{0.31}$
GPT-3.5-Turbo	✗	3	$0.12$	$0.18$	$0.10$	$\bm{0.13}$	$\bm{0.13}$	$\bm{0.15}$	$0.06$	$0.04$	$0.13$	$0.21$	$0.21$	$0.25$
GPT-3.5-Turbo	✗	4	$0.08$	$0.06$	$\bm{0.12}$	$0.12$	$0.11$	$\bm{0.15}$	$0.04$	$0.02$	$0.15$	$0.23$	$0.22$	$0.27$
GPT-3.5-Turbo	✗	5	$0.10$	$0.17$	$0.07$	$0.05$	$0.05$	$0.05$	$\bm{0.16}$	$\bm{0.45}$	$0.10$	$0.12$	$0.12$	$0.14$
GPT-4	✗	1	$0.22$	$0.29$	$0.18$	$0.26$	$0.27$	$0.28$	$\underline{\bm{0.35}}$	$0.81$	$0.22$	$\bm{0.40}$	$\bm{0.40}$	$\bm{0.42}$
GPT-4	✗	2	$\bm{0.34}$	$\bm{0.68}$	$\underline{\bm{0.23}}$	$\bm{0.28}$	$\bm{0.29}$	$\bm{0.30}$	$0.31$	$\bm{0.87}$	$0.19$	$0.39$	$\bm{0.40}$	$0.40$
GPT-4	✗	3	$0.22$	$0.25$	$0.20$	$0.26$	$0.27$	$0.28$	$0.12$	$0.09$	$\bm{0.21}$	$0.39$	$\bm{0.40}$	$\bm{0.42}$
GPT-4	✗	4	$0.32$	$0.65$	$0.21$	$0.25$	$0.26$	$0.29$	$0.12$	$0.08$	$0.20$	$0.38$	$0.38$	$0.40$
GPT-4	✗	5	$0.19$	$0.17$	$0.21$	$0.20$	$0.19$	$0.25$	$0.24$	$0.32$	$0.20$	$0.26$	$0.26$	$0.29$
GPT-3.5-Turbo	✓	1	$0.28$	$0.81$	$0.17$	$0.24$	$0.24$	$0.25$	$0.22$	$0.28$	$0.18$	$\underline{\bm{0.42}}$	$\underline{\bm{0.43}}$	$0.42$
GPT-3.5-Turbo	✓	2	$\underline{\bm{0.35}}$	$0.88$	$\bm{0.22}$	$0.24$	$0.25$	$0.25$	$0.11$	$0.08$	$0.16$	$0.40$	$0.42$	$0.42$
GPT-3.5-Turbo	✓	3	$0.26$	$0.50$	$0.18$	$\bm{0.25}$	$\bm{0.26}$	$\bm{0.26}$	$0.09$	$0.06$	$0.15$	$0.37$	$0.37$	$0.40$
GPT-3.5-Turbo	✓	4	$\underline{\bm{0.35}}$	$0.91$	$\bm{0.22}$	$0.23$	$0.24$	$0.24$	$0.10$	$0.07$	$\bm{0.21}$	$0.41$	$0.41$	$\underline{\bm{0.44}}$
GPT-3.5-Turbo	✓	5	$0.34$	$\underline{\bm{0.92}}$	$0.21$	$0.24$	$0.25$	$0.25$	$\bm{0.31}$	$\bm{0.87}$	$0.19$	$0.38$	$0.38$	$0.40$