notesum.ai

Published at December 3

Patent-CR: A Dataset for Patent Claim Revision

cs.CL

Released Date: December 3, 2024

Authors: Lekang Jiang¹, Pascal A Scherz², Stephan Goetz¹

Aff.: ¹University of Cambridge; ²PSPB Patent Law

Arxiv: http://arxiv.org/pdf/2412.02549v1

Refer to caption

Model	Automated Evaluation					Human Evaluation
Model	SARI	BLEU	R-L	BERTScore	G-Eval	Completeness	Clarity	Consistency	Linkage	Quality
Copy	59.9	0.63	0.68	0.92	80.7	5.67	5.50	5.83	5.33	5.58
CoEdIT-XL	34.6	0.59	0.64	0.91	76.8	5.17	4.82	5.16	4.67	4.97
SaulLM-7B	42.6	0.51	0.61	0.91	81.8	5.50	5.50	5.83	5.50	5.56
SaulLM-7B-FT	55.1	0.63	0.67	0.92	80.7	6.33	6.50	6.67	6.17	6.38
Mixtral-8x7B	33.2	0.27	0.47	0.88	81.7	5.33	5.17	5.67	5.17	5.32
Llama-3.1-8B	38.4	0.48	0.54	0.90	79.4	5.33	5.33	5.17	5.17	5.26
Llama-3.1-8B-FT	55.5	0.62	0.66	0.92	80.3	5.83	6.17	6.33	6.00	6.03
Llama-3.1-70B	38.7	0.49	0.56	0.90	78.1	5.83	5.67	5.83	5.17	5.62
GPT-3.5	38.2	0.49	0.60	0.90	76.9	5.67	5.67	5.83	5.33	5.60
GPT-4	33.7	0.45	0.55	0.89	-	6.67	6.17	6.17	6.33	6.40