notesum.ai

Published at November 11

LongSafetyBench: Long-Context LLMs Struggle with Safety Issues

cs.CL

cs.AI

cs.LG

Released Date: November 11, 2024

Authors: Mianqiu Huang¹, Xiaoran Liu², Shaojun Zhou¹, Mozhi Zhang¹, Chenkun Tan¹, Pengyu Wang¹, Qipeng Guo³, Zhe Xu¹, Linyang Li³, Zhikai Lei³, Linlin Li⁴, Qun Liu⁴, Yaqian Zhou¹, Xipeng Qiu¹, Xuanjing Huang¹

Aff.: ¹School of Computer Science, Fudan University; ²School of Computer Science, Fudan University / Shanghai AI Lab; ³Shanghai AI Lab; ⁴Huawei Noah's Ark Lab

Arxiv: http://arxiv.org/abs/2411.06899v1

Model Name	HE	HT	MJ	HN	CC	DA	HA	MQ	DA	LQ	Avg
Harm Awareness
GPT-4-turbo	0.49	0.81	0.91	0.33	0.47	0.83	0.84	0.88	0.48	0.75	0.68
Qwen2-72b-Instruct	0.38	0.78	0.89	0.15	0.52	0.87	0.96	0.78	0.23	0.76	0.63
Claude-3.5-sonnet	0.52	0.72	0.91	0.41	0.63	0.87	0.93	0.74	0.53	0.96	0.72
Gemini-1.5-pro	0.33	0.57	0.62	0.33	0.56	0.80	0.93	0.86	0.25	0.70	0.60
Llama3.1-70b-Instruct	0.44	0.86	0.83	0.82	1.00	0.78	0.91	0.73	0.24	0.81	0.74
Llama3.1-8b-Instruct	0.42	0.64	0.68	0.80	0.95	0.80	0.82	0.59	0.21	0.66	0.66
Mistral-7b-Instruct-v0.3	0.48	0.56	0.73	0.63	0.92	0.89	0.58	0.59	0.17	0.64	0.62
Mixtral8x22B-Instruct-v0.1	0.45	0.69	0.61	0.53	0.96	0.59	0.40	0.38	0.47	0.53	0.56
GLM-4-9b-chat	0.49	0.54	0.28	0.42	0.56	0.43	0.47	0.01	0.54	0.43	0.42
InternLM2.5-7b-chat	0.18	0.47	0.95	0.29	0.61	0.78	0.91	0.66	0.21	0.66	0.57
Safe Response
GPT-4-turbo	0.33	0.63	0.78	0.32	0.24	0.52	0.73	0.88	0.48	0.56	0.55
Qwen2-72b-Instruct	0.09	0.51	0.66	0.15	0.20	0.76	0.87	0.78	0.23	0.41	0.47
Claude-3.5-sonnet	0.33	0.69	0.76	0.41	0.35	0.70	0.78	0.74	0.53	0.91	0.62
Gemini-1.5-pro	0.06	0.40	0.34	0.33	0.19	0.39	0.56	0.86	0.25	0.55	0.39
Llama3.1-70b-Instruct	0.11	0.67	0.66	0.41	0.46	0.52	0.67	0.73	0.24	0.54	0.50
Llama3.1-8b-Instruct	0.08	0.45	0.45	0.65	0.26	0.50	0.64	0.59	0.21	0.37	0.42
Mistral-7b-Instruct-v0.3	0.03	0.21	0.44	0.57	0.40	0.70	0.40	0.59	0.17	0.21	0.37
Mixtral8x22B-Instruct-v0.1	0.18	0.39	0.36	0.36	0.22	0.30	0.11	0.38	0.47	0.23	0.30
GLM-4-9b-chat	0.30	0.30	0.15	0.22	0.39	0.20	0.29	0.01	0.54	0.29	0.27
InternLM2.5-7b-chat	0.05	0.28	0.81	0.07	0.23	0.70	0.84	0.66	0.21	0.42	0.43