notesum.ai

Published at December 3

cs.SE

cs.AI

Released Date: December 3, 2024

Authors: Donghao Yang¹, Aolang Wu, Tianyi Zhang, Li Zhang, Fang Liu, Xiaoli Lian, Yuming Ren, Jiaji Tian

Aff.: ¹Beihang University, China

Baselines	OSCAT (CODESYS’ ST)		LGF (Siemens’ SCL)		Competition (Siemens’ SCL)
Baselines	Passed Ratio	Avg. Errors	Passed Ratio	Avg. Errors	Passed Ratio	Avg. Errors
LLM4PLC	30.74%	15.52	7.28%	16.54	9.09%	21.80
Llama3.1	5.98%	28.14	0.66%	15.65	0.00%	22.84
Qwen2.5	18.08%	13.14	5.96%	10.92	5.61%	9.56
GLM-4-plus	35.05%	12.22	6.62%	10.54	6.67%	9.36
GPT-4o	35.47%	8.39	8.61%	18.76	20.00%	13.49
DeepSeek-V2.5	42.70%	7.68	3.97%	14.53	22.22%	13.27
Claude-3.5	57.72%	5.76	15.23%	9.60	26.67%	6.53
AutoPLC	72.26%	3.62	58.94%	3.60	60.00%	2.64