notesum.ai

Published at November 21

OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

cs.CL

cs.AI

cs.IR

Released Date: November 21, 2024

Authors: Akari Asai¹, Jacqueline He¹, Rulin Shao¹, Weijia Shi¹, Amanpreet Singh², Joseph Chee Chang², Kyle Lo², Luca Soldaini², Sergey Feldman², Mike D'arcy², David Wadden², Matt Latzke², Minyang Tian³, Pan Ji⁴, Shengyan Liu³, Hao Tong³, Bohao Wu³, Yanyu Xiong⁵, Luke Zettlemoyer¹, Graham Neubig⁶, Dan Weld¹, Doug Downey², Wen-tau Yih⁷, Pang Wei Koh¹, Hannaneh Hajishirzi¹

Aff.: ¹University of Washington; ²Allen Institute for AI; ³University of Illinois, Urbana-Champaign; ⁴University of North Carolina, Chapel Hill; ⁵Stanford University; ⁶Carnegie Mellon University; ⁷Meta

Arxiv: http://arxiv.org/abs/2411.14199v1

Dataset	Task Format	Discipline	Size	Evaluation	Multi-paper
SciFact	Claim $\rightarrow$ Label	Biomedicine	208	,
(Wadden et al. 2020)	(True or False)
PubMed QA	Question $\rightarrow$ Answer	Biomedicine	843	,
(Jin et al. 2019)	(Yes, No)
QASA	Question $\rightarrow$ Answer	Computer Science	1,375	,
(Lee et al. 2023)	(Long-form)
ScholarQA-CS	Question $\rightarrow$ Answer^†	Computer Science	100	,	✓
	(Long-form)				✓
ScholarQA-Bio	Question $\rightarrow$ Answer^∗	Biomedicine	1,451		✓
	(Long-form)				✓
ScholarQA-Neuro	Question $\rightarrow$ Answer^∗	Neuroscience	1,308		✓
	(Long-form)				✓
ScholarQA-Multi	Question $\rightarrow$ Answer	Computer Science, Physics,	108	,	✓
	(Long-form)	Biomedicine		,	✓