notesum.ai

Published at November 9

Smart-LLaMA: Two-Stage Post-Training of Large Language Models for Smart Contract Vulnerability Detection and Explanation

cs.CR

cs.AI

cs.SE

Released Date: November 9, 2024

Authors: Lei Yu¹, Shiqi Chen, Hang Yuan¹, Peng Wang¹, Zhirong Huang¹, Jingyuan Zhang¹, Chenjie Shen¹, Fengjun Zhang², Li Yang², Jiajia Ma²

Aff.: ¹Institute of Software, Chinese Academy of Sciences, Beijing, China; ²University of Chinese Academy of Sciences, Beijing, China

Arxiv: http://arxiv.org/abs/2411.06221v1

Methods	Reentrancy				Timestamp Dependency				Overflow/Underflow				Delegatecall
	A(%)	P(%)	R(%)	F1(%)	A(%)	P(%)	R(%)	F1(%)	A(%)	P(%)	R(%)	F1(%)	A(%)	P(%)	R(%)	F1(%)
Mythril	54.21	33.75	73.97	46.35	43.27	44.63	44.13	44.38	37.45	25.30	46.67	32.81	60.71	42.99	74.19	54.44
Osiris	30.77	26.80	91.78	41.49	51.03	52.50	36.63	43.15	61.90	45.33	75.56	56.67	–	–	–	–
Oyente	68.50	42.35	49.32	45.57	53.98	66.67	18.60	29.09	72.53	60.87	46.67	52.83	64.71	40.00	31.58	35.29
Slither	37.73	16.08	31.51	21.30	57.02	56.50	70.39	62.69	50.91	32.28	45.56	37.79	52.04	39.04	91.94	54.81
Smartcheck	43.22	30.10	84.93	44.44	51.00	57.14	17.88	27.23	52.00	31.25	38.89	34.65	54.08	32.93	43.55	37.50
GCN	73.21	74.47	73.18	73.82	75.91	74.93	77.55	76.22	67.53	69.52	70.93	70.22	65.76	69.01	69.74	69.37
TMP	76.45	76.04	75.30	75.67	78.84	78.68	76.09	77.36	70.85	70.26	69.47	69.86	69.11	68.18	70.37	69.26
AME	81.06	79.62	78.45	79.03	82.25	81.42	80.26	80.84	73.24	71.36	71.59	71.47	72.85	70.25	69.40	69.82
SMS	83.85	79.46	77.48	78.46	89.77	89.15	91.09	90.11	79.36	78.14	72.98	75.47	78.82	76.97	73.69	75.29
DMT	89.42	83.62	81.06	82.32	94.58	93.60	96.39	94.97	85.64	85.44	74.32	79.49	82.76	84.61	77.93	81.13
Peculiar	58.72	35.23	48.44	40.84	69.08	77.31	71.12	74.18	75.91	64.29	60.00	62.07	88.46	77.36	82.00	79.61
PSCVFinder	58.26	35.16	50.00	41.32	40.10	52.00	50.39	51.18	51.09	33.58	50.00	40.18	89.56	79.25	84.00	81.57
LLaMA-3.1-8B	36.24	29.73	85.94	44.18	57.00	67.24	60.47	63.67	55.11	38.93	64.44	48.54	63.74	38.89	56.00	45.90
Qwen2-7B	30.28	29.63	100.00	45.71	67.63	71.83	79.07	75.28	66.42	47.62	22.22	30.30	65.93	44.23	92.00	59.74
LLaMA-3.1-70B	29.36	29.36	100.00	45.39	62.32	62.32	100.00	76.79	77.74	64.08	73.33	68.39	65.93	44.55	98.00	61.25
Qwen2-72B	35.78	31.37	100.00	47.76	69.08	67.38	97.67	79.75	75.55	64.94	55.56	59.88	74.73	52.08	100.00	68.49
Smart-LLaMA	93.12	87.69	89.06	88.37	95.17	95.42	96.90	96.15	89.78	79.25	93.33	85.71	94.51	95.45	84.00	89.36