notesum.ai

Published at December 10

Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs without Real Data Replay

cs.CL

cs.DB

Released Date: December 10, 2024

Authors: Ruiheng Liu¹, Jinyu Zhang², Yanqi Song², Yu Zhang², Bailong Yang¹

Aff.: ¹Xi'an Research Institute of High-Tech, Xi'an, China; ²Harbin Institute of Technology, Harbin, China

Arxiv: http://arxiv.org/pdf/2412.07246v1

Method	DR	Spider-stream-semi					Combined-stream
Method	DR	$\text{ACC}_{\text{a}}$	$\text{ACC}_{\text{w}}$	BWT	FWT	$\Delta$	$\text{ACC}_{\text{a}}$	$\text{ACC}_{\text{w}}$	BWT	FWT	$\Delta$
T5-Base(220M)
SFT	✗	41.7/45.3	40.1/43.7	-13.6/-11.7	22.5/25.2	$\downarrow$ 1.5	51.0/36.0	52.4/38.3	-11.1/-17.6	19.9/11.1	$\downarrow$ 11.1
ProgPrompt	✗	14.9/16.5	14.3/16.4	-25.6/-26.3	14.7/15.8	$\downarrow$ 14.9	21.3/19.2	27.2/26.8	-43.9/-41.2	6.1/3.3	$\downarrow$ 21.1
EMAR	✓	44.1/48.0	43.0/47.0	-11.8/-9.7	23.7/25.2	$\uparrow$ 2.3	60.6/58.9	55.7/55.3	-7.9/-7.1	28.7/25.4	$\downarrow$ 5.6
SFNET^♣	✗	39.5/43.9	38.1/42.7	-14.6/-12.4	22.7/25.7	$\uparrow$ 0.4	56.2/49.4	51.8/48.4	-7.9/-14.4	21.0/14.6	$\downarrow$ 21.7
SFNET^♣	✓	45.8/48.1	44.2/47.7	-6.4/-6.4	23.9/26.2	$\downarrow$ 3.4	60.7/57.7	55.2/53.9	-6.0/-6.7	28.7/26.0	$\downarrow$ 6.7
Lecsp(Ours)	✗	47.4/53.7	46.5/53.3	-4.7/-4.6	30.1/32.8	$\uparrow$ 1.4	63.4/61.7	60.5/59.8	-4.6/4.3	32.7/31.5	$\downarrow$ 4.8
Ideal Setting & Upper Bound
PEFT^♢	✗	40.6/43.8	44.6/48.4	0.0/0.0	-/-	$\downarrow$ 32.5	63.8/-	66.2/-	0.0/0.0	-/-	$\downarrow$ 38.8
ORACLE		60.3/61.3	62.8/63.9	4.6/3.3	25.7/27.9	$\downarrow$ 1.1	70.2/68.2	71.1/70.8	5.6/4.7	29.2/26.6	$\uparrow$ 0.3
T5-Large(770M)
SFT	✗	46.4/50.4	44.3/48.9	-13.2/-11.5	29.0/31.5	$\uparrow$ 1.8	57.8/48.9	58.9/53.3	-7.6/-16.4	28.3/25.4	$\downarrow$ 8.3
ProgPrompt	✗	22.1/23.9	22.4/24.6	-27.0/-27.5	21.1/23.1	$\downarrow$ 20.8	23.5/20.2	29.8/27.7	-46.4/-42.7	10.2/3.7	$\downarrow$ 22.4
EMAR	✓	48.9/51.8	48.4/52.5	-8.5/-8.3	30.3/33.5	$\uparrow$ 1.4	63.9/61.9	60.6/60.4	-6.6/-8.9	33.8/29.3	$\downarrow$ 5.5
SFNET^♣	✗	44.2/48.2	43.3/47.8	-12.3/-9.9	29.9/31.0	$\uparrow$ 3.3	65.0/60.3	61.5/59.7	-7.5/-11.9	28.2/20.4	$\downarrow$ 24.1
SFNET^♣	✓	52.2/55.0	52.0/55.8	-6.5/-4.2	34.1/35.5	$\downarrow$ 2.2	65.3/64.7	61.8/61.9	-5.7/-5.2	34.9/31.1	$\downarrow$ 5.6
Lecsp(Ours)	✗	58.6/63.5	57.4/63.8	-5.8/-5.0	41.4/44.3	$\uparrow$ 0.8	68.2/66.7	66.5/65.4	-2.9/-3.7	37.1/34.5	$\downarrow$ 5.1
Ideal Setting & LLMs Performance & Upper Bound
PEFT^♢	✗	49.6/52.4	53.4/56.4	0.0/0.0	-/-	$\downarrow$ 24.1	67.3/-	70.0/-	0.0/0.0	-/-	$\downarrow$ 23.4
C3^♢	✗	-/-	-/-	-/-	-/-	-	69.0/-	71.2/-	0.0/0.0	-/-	-
C3^♢†	✗	-/-	-/-	-/-	-/-	-	67.6/-	70.0/-	0.0/0.0	-/-	-
Mixtral-8x7B	✗	22.0/49.4	22.2/52.4	-/-	-/-	-	28.6/24.4	27.0/26.0	-/-	-/-	-
ORACLE		66.6/68.2	68.6/70.1	3.7/4.0	34.3/36.0	$\uparrow$ 0.4	73.7/73.2	75.8/76.0	1.9/2.3	37.3/34.2	$\downarrow$ 0.1