notesum.ai

Published at October 24

Context is Key: A Benchmark for Forecasting with Essential Textual Information

cs.HC

cs.AI

Released Date: October 24, 2024

Authors: Andrew Robert Williams, Arjun Ashok¹, Étienne Marcotte, Valentina Zantedeschi², Jithendaraa Subramanian³, Roland Riachi⁴, James Requeima⁵, Alexandre Lacoste⁶, Irina Rish⁷, Nicolas Chapados⁸, Alexandre Drouin⁹

Aff.: ¹ServiceNow Research, Mila - Quebec AI Institute, Universite de Montreal; ²ServiceNow Research, Universite Laval; ³ServiceNow Research, Mila - Quebec AI Institute, McGill University; ⁴Mila - Quebec AI Institute; ⁵University of Toronto; ⁶ServiceNow Research; ⁷Mila - Quebec AI Institute, Universite de Montreal; ⁸ServiceNow Research, Mila - Quebec AI Institute, Polytechnique Montreal; ⁹ServiceNow Research, Mila - Quebec AI Institute, Universite Laval

Arxiv: https://arxiv.org/abs/2410.18959v1

	Average RCRPS	Average Rank	Instruction Following	Retrieval		Reasoning
Model	Average RCRPS	Average Rank	Instruction Following	From Context	From Memory	Deductive	Analogical	Mathematical	Causal
Direct Prompt (ours)
Llama-3.1-405B-Inst	0.159 $\pm$ 0.008	4.469 $\pm$ 0.192	0.140 $\pm$ 0.013	0.109 $\pm$ 0.002	0.191 $\pm$ 0.006	0.133 $\pm$ 0.001	0.167 $\pm$ 0.008	0.316 $\pm$ 0.028	0.376 $\pm$ 0.039
Llama-3-70B-Inst	0.518 $\pm$ 0.030	10.406 $\pm$ 0.190	0.504 $\pm$ 0.038	0.371 $\pm$ 0.071	0.523 $\pm$ 0.048	0.461 $\pm$ 0.048	0.694 $\pm$ 0.117	0.573 $\pm$ 0.044	0.643 $\pm$ 0.049
Llama-3-8B-Inst	1.647 $\pm$ 0.069	15.130 $\pm$ 0.171	1.604 $\pm$ 0.131	0.199 $\pm$ 0.010	1.568 $\pm$ 0.067	2.133 $\pm$ 0.082	1.555 $\pm$ 0.008	1.589 $\pm$ 0.177	1.840 $\pm$ 0.238
Mixtral-8x7B-Inst	1.061 $\pm$ 0.058	13.381 $\pm$ 0.230	0.857 $\pm$ 0.077	0.296 $\pm$ 0.049	1.077 $\pm$ 0.078	1.352 $\pm$ 0.117	1.145 $\pm$ 0.144	1.000 $\pm$ 0.086	1.096 $\pm$ 0.106
GPT-4o	0.276 $\pm$ 0.010	4.368 $\pm$ 0.149	0.180 $\pm$ 0.004	0.087 $\pm$ 0.003	0.519 $\pm$ 0.029	0.113 $\pm$ 0.006	0.447 $\pm$ 0.029	0.590 $\pm$ 0.033	0.769 $\pm$ 0.046
GPT-4o-mini	0.353 $\pm$ 0.022	8.930 $\pm$ 0.177	0.296 $\pm$ 0.043	0.419 $\pm$ 0.014	0.471 $\pm$ 0.012	0.218 $\pm$ 0.005	1.024 $\pm$ 0.033	0.475 $\pm$ 0.080	0.578 $\pm$ 0.112
LLMP
Llama-3-70B-Inst	0.550 $\pm$ 0.013	8.038 $\pm$ 0.205	0.645 $\pm$ 0.018	0.284 $\pm$ 0.015	0.392 $\pm$ 0.014	0.519 $\pm$ 0.026	0.312 $\pm$ 0.019	0.453 $\pm$ 0.020	0.495 $\pm$ 0.028
Llama-3-70B	0.237 $\pm$ 0.006	6.560 $\pm$ 0.254	0.310 $\pm$ 0.011	0.126 $\pm$ 0.009	0.217 $\pm$ 0.007	0.134 $\pm$ 0.003	0.241 $\pm$ 0.019	0.294 $\pm$ 0.008	0.329 $\pm$ 0.010
Llama-3-8B-Inst	0.484 $\pm$ 0.010	9.457 $\pm$ 0.166	0.345 $\pm$ 0.002	0.138 $\pm$ 0.004	0.910 $\pm$ 0.030	0.242 $\pm$ 0.008	1.278 $\pm$ 0.069	0.617 $\pm$ 0.022	0.787 $\pm$ 0.030
Llama-3-8B	0.313 $\pm$ 0.023	9.499 $\pm$ 0.323	0.404 $\pm$ 0.043	0.124 $\pm$ 0.003	0.280 $\pm$ 0.026	0.179 $\pm$ 0.014	0.267 $\pm$ 0.015	0.530 $\pm$ 0.084	0.661 $\pm$ 0.117
Mixtral-8x7B-Inst	0.264 $\pm$ 0.004	8.496 $\pm$ 0.256	0.344 $\pm$ 0.004	0.127 $\pm$ 0.003	0.224 $\pm$ 0.005	0.179 $\pm$ 0.010	0.173 $\pm$ 0.009	0.348 $\pm$ 0.005	0.405 $\pm$ 0.007
Mixtral-8x7B	0.262 $\pm$ 0.008	8.619 $\pm$ 0.208	0.348 $\pm$ 0.012	0.146 $\pm$ 0.022	0.230 $\pm$ 0.016	0.153 $\pm$ 0.002	0.230 $\pm$ 0.041	0.354 $\pm$ 0.007	0.414 $\pm$ 0.009
Multimodal Models
UniTime	0.371 $\pm$ 0.002	13.495 $\pm$ 0.091	0.271 $\pm$ 0.003	0.179 $\pm$ 0.001	0.318 $\pm$ 0.001	0.510 $\pm$ 0.003	0.333 $\pm$ 0.001	0.332 $\pm$ 0.001	0.384 $\pm$ 0.001
Time-LLM (ETTh1)	0.476 $\pm$ 0.001	16.662 $\pm$ 0.075	0.448 $\pm$ 0.002	0.192 $\pm$ 0.000	0.373 $\pm$ 0.000	0.538 $\pm$ 0.003	0.397 $\pm$ 0.001	0.382 $\pm$ 0.001	0.440 $\pm$ 0.001
TS Foundation Models*
Lag-Llama	0.329 $\pm$ 0.004	13.157 $\pm$ 0.235	0.355 $\pm$ 0.007	0.181 $\pm$ 0.003	0.324 $\pm$ 0.003	0.272 $\pm$ 0.006	0.342 $\pm$ 0.006	0.386 $\pm$ 0.009	0.449 $\pm$ 0.012
Chronos	0.326 $\pm$ 0.002	11.962 $\pm$ 0.139	0.385 $\pm$ 0.002	0.138 $\pm$ 0.002	0.288 $\pm$ 0.002	0.249 $\pm$ 0.002	0.295 $\pm$ 0.003	0.362 $\pm$ 0.003	0.417 $\pm$ 0.004
TimeGEN	0.354 $\pm$ 0.000	14.345 $\pm$ 0.090	0.402 $\pm$ 0.000	0.176 $\pm$ 0.000	0.308 $\pm$ 0.000	0.279 $\pm$ 0.000	0.324 $\pm$ 0.000	0.377 $\pm$ 0.000	0.431 $\pm$ 0.000
Moirai	0.520 $\pm$ 0.006	12.458 $\pm$ 0.266	0.414 $\pm$ 0.004	0.155 $\pm$ 0.004	0.260 $\pm$ 0.003	0.751 $\pm$ 0.015	0.276 $\pm$ 0.008	0.337 $\pm$ 0.007	0.397 $\pm$ 0.010
Statistical Models*
ARIMA	0.480 $\pm$ 0.006	12.320 $\pm$ 0.180	0.399 $\pm$ 0.006	0.160 $\pm$ 0.002	0.517 $\pm$ 0.012	0.522 $\pm$ 0.013	0.706 $\pm$ 0.026	0.354 $\pm$ 0.007	0.403 $\pm$ 0.010
ETS	0.522 $\pm$ 0.009	14.310 $\pm$ 0.203	0.407 $\pm$ 0.009	0.228 $\pm$ 0.010	0.682 $\pm$ 0.018	0.571 $\pm$ 0.019	0.855 $\pm$ 0.035	0.453 $\pm$ 0.012	0.479 $\pm$ 0.015
Exp-Smoothing	0.603 $\pm$ 0.013	14.936 $\pm$ 0.132	0.571 $\pm$ 0.021	0.334 $\pm$ 0.013	0.743 $\pm$ 0.018	0.557 $\pm$ 0.019	0.899 $\pm$ 0.035	0.673 $\pm$ 0.038	0.782 $\pm$ 0.053