notesum.ai

Published at November 7

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

cs.AI

cs.MA

Released Date: November 7, 2024

Authors: Adam Fourney¹, Gagan Bansal¹, Hussein Mozannar¹, Cheng Tan¹, Eduardo Salinas¹, Erkang, Zhu, Friederike Niedtner¹, Grace Proebsting¹, Griffin Bassman¹, Jack Gerrits¹, Jacob Alber¹, Peter Chang¹, Ricky Loynd¹, Robert West¹, Victor Dibia¹, Ahmed Awadallah¹, Ece Kamar¹, Rafah Hosn¹, Saleema Amershi¹

Aff.: ¹Microsoft Research

Arxiv: http://arxiv.org/abs/2411.04468v1

Method	GAIA	AssistantBench (EM)	AssistantBench (accuracy)	WebArena
omne v0.1 (GPT-4o, o1)	40.53 $\pm$ 5.6	–	–	–
Trase Agent v0.2 (GPT-4o, o1, Gemini)	39.53 $\pm$ 5.5	–	–	–
Multi Agent (NA)	38.87 $\pm$ 5.5	–	–	–
das agent v0.4 (GPT-4o)	38.21 $\pm$ 5.5	–	–	–
Sibyl (GPT-4o) [56]	34.55 $\pm$ 5.4	–	–	–
HF Agents (GPT-4o)	33.33 $\pm$ 5.3	–	–	–
FRIDAY (GPT-4T) [61]	24.25 $\pm$ 4.8	–	–	–
GPT-4 + plugins [29]	14.60 $\pm$ 4.0	–	–	–
SPA $\rightarrow$ CB (Claude) [71]	–	13.8 $\pm$ 5.0	26.4 $\pm$ 6.4	–
SPA $\rightarrow$ CB (GPT-4T) [71]	–	9.9 $\pm$ 4.3	25.2 $\pm$ 6.3	–
Infogent (GPT-4o)	–	5.5 $\pm$ 3.3	14.5 $\pm$ 5.1	–
Jace.AI (NA)	–	–	–	57.1 $\pm$ 3.4
WebPilot (GPT-4o) [75]	–	–	–	37.2 $\pm$ 3.3
AWM (GPT-4) [57]	–	–	–	35.5 $\pm$ 3.3
SteP (GPT-4) [49]	–	–	–	33.5 $\pm$ 3.2
BrowserGym (GPT-4o) [10]	–	–	–	23.5 $\pm$ 2.9
GPT-4	6.67 $\pm$ 2.8[29]	6.1 $\pm$ 3.5[71]	16.5 $\pm$ 5.4[71]	14.9 $\pm$ 2.4[79]
Human	92.00 $\pm$ 3.1	–	–	78.2 $\pm$ 2.8
Magentic-One (GPT-4o)	32.33 $\pm$ 5.3	11.0 $\pm$ 4.6	25.3 $\pm$ 6.3	32.8 $\pm$ 3.2
Magentic-One (GPT-4o, o1)	38.00 $\pm$ 5.5	13.3 $\pm$ 4.9	27.7 $\pm$ 6.5	*