notesum.ai

Published at November 28

DENIAHL: In-Context Features Influence LLM Needle-In-A-Haystack Abilities

cs.CL

cs.AI

cs.LG

Released Date: November 28, 2024

Authors: Hui Dai¹, Dan Pechi², Xinyi Yang¹, Garvit Banga¹, Raghav Mantri¹

Aff.: ¹New York University; ²Databricks

Arxiv: http://arxiv.org/pdf/2411.19360v1

Refer to caption

Task	Configuration	Example	Key	Gold Label
Data Size	Number of items: varying the number of key-value pairs	number of items = 3 { "e3c90162-c33e-46": "113007f3-4421-42", "05b17449-c3d6-4b": "76231614-9b70-4f", "bb7a3f7e-55a7-45": "22ef5b80-1656-40"}	05b17449-c3d6-4b	76231614-9b70-4f
	Item length: varying the length of the keys/values	item length = 8 { "66b1b1c9": "d53e0cf5", … "6669d056": "cae5c7ee" }	66b1b1c9	d53e0cf5
Pattern	Numerical pattern Letter pattern	{ 1: 3, 2: 5, 3: 6, 4: 9, 5: 11} { "xf": "XF", "gf": "GF", "pr": "AB", "pl": "PL"} *Note that we break the pattern of the value for the corresponding key we want to retrieve	3 pr	6 AB
Data Type	All letters All numbers Mixed	{ "aBc": "xYz", "Aaa": "sdD", "xcL": "AQw" } { 079: 234, 194: 566, 243: 746 } { "a03xcl": "fef4t5", "Qs235r": "2ref63"}	xcL 194 a03xcl	AQw 566 fef4t5