notesum.ai

Published at November 6

MEG: Medical Knowledge-Augmented Large Language Models for Question Answering

cs.CL

cs.AI

cs.LG

Released Date: November 6, 2024

Authors: Laura Cabello¹, Carmen Martin-Turrero¹, Uchenna Akujuobi¹, Anders Søgaard², Carlos Bobed³

Aff.: ¹Sony AI, Barcelona, Spain; ²University of Copenhagen, Denmark; ³University of Zaragoza, Spain

Arxiv: http://arxiv.org/abs/2411.03883v1

MedQA

PubMedQA

MedMCQA

MMLU-Medical

Avg

MMLU-Medical

Human (pass)

60.0

Clinical K.

Genetics

Anatomy

P. Medicine

C. Biology

C. Medicine

Human (expert)

87.0

78.0

90.0

Models based on Llama

MedAlpaca (7B)^†

40.1_±0.4

73.6_±0.3

37.0_±0.3

55.1_±1.1

51.4

53.1_±0.9

58.0_±2.2

54.1_±1.6

58.8_±0.3

58.1_±1.3

48.6_±0.5

MEDITRON-7B^∗‡

52.0_±-

74.4_±-

59.2_±-

54.2_±-

60.0

57.2_±-

64.6_±-

49.3_±-

55.4_±-

53.8_±-

44.8_±-

Meerkat-8B^∗‡

74.2_±-

62.7_±-

75.2_±-

70.7

74.3_±-

76.7_±-

74.8_±-

75.3_±-

76.1_±-

74.3_±-

MEG-Llama (8B)

66.0_±0.2

78.0_±0.3

60.6_±0.3

74.9_±0.7

69.9

72.3_±0.5

83.0_±1.5

64.5_±0.7

79.4_±0.3

80.6_±0.4

69.4_±0.9

Models based on Mistral-Instruct 7B

Mistral-Instruct-v0.1^†

42.0_±0.2

73.8

{}^{\triangledown}_{\pm{0.4}}

46.1_±0.1

59.1_±1.0

55.3

62.9_±0.2

57.0_±0.8

55.6_±1.0

59.4_±0.6

62.5_±1.0

57.2_±2.1

BioMistral^†

50.6_±0.3

77.5_±0.1

48.1_±0.2

59.1_±1.3

58.8

59.9_±1.2

64.0_±1.6

56.5_±1.8

60.4_±0.5

59.0_±1.5

54.7_±1.0

BioMistral DARE^†

51.1_±0.3

77.7_±0.1

48.7_±0.1

61.9_±1.2

59.9

62.3_±1.3

67.0_±1.6

55.8_±0.9

61.4_±0.3

66.9_±2.3

58.0_±0.5

Meerkat-7B ^‡

70.3_±-

60.6_±-

70.5_±-

67.1

71.6_±-

74.8_±-

63.2_±-

77.3_±-

70.8_±-

65.2_±-

MEG-Mistral1

54.6_±0.2

74.6_±0.6

56.4_±0.4

60.3_±0.9

61.5

58.1_±0.8

68.7_±0.2

54.4_±0.5

62.9_±0.9

61.1_±2.2

56.6_±1.0

MEG-Mistral3

60.8_±0.2

74.4_±0.5

58.4_±0.6

68.2_±0.4

65.5

64.9_±0.2

69.6_±0.8

63.0_±1.0

72.8_±0.4

73.6_±0.0

65.2_±0.2