Hindsight/Benchmarks
CommunityDocsGitHub
← Back

LongMemEval Benchmark

Long-term memory evaluation across extended conversations

Overall

Accuracy

91.40%

Correct

457 / 500

Items

500

By Category

knowledge-update

94.9%

74 / 78

multi-session

87.2%

116 / 133

single-session-assistant

96.4%

54 / 56

single-session-preference

80.0%

24 / 30

single-session-user

97.1%

68 / 70

temporal-reasoning

91.0%

121 / 133

Items (500)

Showing 500 of 500 items

e47becba

100.0%1/1 correct

118b2229

100.0%1/1 correct

51a45a95

100.0%1/1 correct

58bf7951

100.0%1/1 correct

1e043500

100.0%1/1 correct

c5e8278d

100.0%1/1 correct

6ade9755

100.0%1/1 correct

6f9b354f

100.0%1/1 correct

58ef2f1c

100.0%1/1 correct

f8c5f88b

100.0%1/1 correct

5d3d2817

100.0%1/1 correct

7527f7e2

100.0%1/1 correct

c960da58

100.0%1/1 correct

3b6f954b

0.0%0/1 correct

726462e0

100.0%1/1 correct

94f70d80

100.0%1/1 correct

66f24dbb

100.0%1/1 correct

ad7109d1

100.0%1/1 correct

af8d2e46

100.0%1/1 correct

dccbc061

100.0%1/1 correct

c8c3f81d

100.0%1/1 correct

8ebdbe50

100.0%1/1 correct

6b168ec8

100.0%1/1 correct

75499fd8

100.0%1/1 correct

21436231

100.0%1/1 correct

95bcc1c8

100.0%1/1 correct

0862e8bf

100.0%1/1 correct

853b0a1d

100.0%1/1 correct

a06e4cfe

100.0%1/1 correct

37d43f65

100.0%1/1 correct

b86304ba

100.0%1/1 correct

d52b4f67

100.0%1/1 correct

25e5aa4f

100.0%1/1 correct

caf9ead2

100.0%1/1 correct

8550ddae

100.0%1/1 correct

60d45044

100.0%1/1 correct

3f1e9474

100.0%1/1 correct

86b68151

100.0%1/1 correct

577d4d32

100.0%1/1 correct

ec81a493

0.0%0/1 correct

15745da0

100.0%1/1 correct

e01b8e2f

100.0%1/1 correct

bc8a6e93

100.0%1/1 correct

ccb36322

100.0%1/1 correct

001be529

100.0%1/1 correct

b320f3f8

100.0%1/1 correct

19b5f2b3

100.0%1/1 correct

4fd1909e

100.0%1/1 correct

545bd2b5

100.0%1/1 correct

8a137a7f

100.0%1/1 correct

76d63226

100.0%1/1 correct

86f00804

100.0%1/1 correct

8e9d538c

100.0%1/1 correct

311778f1

100.0%1/1 correct

c19f7a0b

100.0%1/1 correct

4100d0a0

100.0%1/1 correct

29f2956b

100.0%1/1 correct

1faac195

100.0%1/1 correct

faba32e5

100.0%1/1 correct

f4f1d8a4

100.0%1/1 correct

c14c00dd

100.0%1/1 correct

36580ce8

100.0%1/1 correct

3d86fd0a

100.0%1/1 correct

a82c026e

100.0%1/1 correct

0862e8bf_abs

100.0%1/1 correct

15745da0_abs

100.0%1/1 correct

bc8a6e93_abs

100.0%1/1 correct

19b5f2b3_abs

100.0%1/1 correct

29f2956b_abs

100.0%1/1 correct

f4f1d8a4_abs

100.0%1/1 correct

0a995998

0.0%0/1 correct

6d550036

0.0%0/1 correct

gpt4_59c863d7

100.0%1/1 correct

b5ef892d

100.0%1/1 correct

e831120c

100.0%1/1 correct

3a704032

0.0%0/1 correct

gpt4_d84a3211

100.0%1/1 correct

aae3761f

100.0%1/1 correct

gpt4_f2262a51

100.0%1/1 correct

dd2973ad

100.0%1/1 correct

c4a1ceb8

100.0%1/1 correct

gpt4_a56e767c

100.0%1/1 correct

6cb6f249

100.0%1/1 correct

46a3abf7

100.0%1/1 correct

36b9f61e

100.0%1/1 correct

28dc39ac

100.0%1/1 correct

gpt4_2f8be40d

0.0%0/1 correct

2e6d26dc

100.0%1/1 correct

gpt4_15e38248

100.0%1/1 correct

88432d0a

0.0%0/1 correct

80ec1f4f

100.0%1/1 correct

d23cf73b

100.0%1/1 correct

gpt4_7fce9456

0.0%0/1 correct

d682f1a2

100.0%1/1 correct

7024f17c

0.0%0/1 correct

gpt4_5501fe77

100.0%1/1 correct

gpt4_2ba83207

0.0%0/1 correct

2318644b

0.0%0/1 correct

2ce6a0f2

100.0%1/1 correct

gpt4_d12ceb0e

100.0%1/1 correct

00ca467f

100.0%1/1 correct

b3c15d39

100.0%1/1 correct

gpt4_31ff4165

100.0%1/1 correct

eeda8a6d

100.0%1/1 correct

2788b940

100.0%1/1 correct

60bf93ed

100.0%1/1 correct

9d25d4e0

0.0%0/1 correct

129d1232

100.0%1/1 correct

60472f9c

100.0%1/1 correct

gpt4_194be4b3

100.0%1/1 correct

a9f6b44c

100.0%1/1 correct

d851d5ba

100.0%1/1 correct

5a7937c8

100.0%1/1 correct

gpt4_ab202e7f

0.0%0/1 correct

gpt4_e05b82a6

0.0%0/1 correct

gpt4_731e37d7

100.0%1/1 correct

edced276

100.0%1/1 correct

10d9b85a

100.0%1/1 correct

e3038f8c

100.0%1/1 correct

2b8f3739

100.0%1/1 correct

1a8a66a6

100.0%1/1 correct

c2ac3c61

100.0%1/1 correct

bf659f65

100.0%1/1 correct

gpt4_372c3eed

100.0%1/1 correct

gpt4_2f91af09

100.0%1/1 correct

81507db6

100.0%1/1 correct

88432d0a_abs

100.0%1/1 correct

80ec1f4f_abs

100.0%1/1 correct

eeda8a6d_abs

100.0%1/1 correct

60bf93ed_abs

100.0%1/1 correct

edced276_abs

100.0%1/1 correct

gpt4_372c3eed_abs

100.0%1/1 correct

8a2466db

100.0%1/1 correct

06878be2

100.0%1/1 correct

75832dbd

0.0%0/1 correct

0edc2aef

0.0%0/1 correct

35a27287

100.0%1/1 correct

32260d93

0.0%0/1 correct

195a1a1b

100.0%1/1 correct

afdc33df

100.0%1/1 correct

caf03d32

100.0%1/1 correct

54026fce

100.0%1/1 correct

06f04340

100.0%1/1 correct

6b7dfb22

100.0%1/1 correct

1a1907b4

100.0%1/1 correct

09d032c9

100.0%1/1 correct

38146c39

100.0%1/1 correct

d24813b1

100.0%1/1 correct

57f827a0

100.0%1/1 correct

95228167

100.0%1/1 correct

505af2f5

100.0%1/1 correct

75f70248

100.0%1/1 correct

d6233ab6

100.0%1/1 correct

1da05512

100.0%1/1 correct

fca70973

0.0%0/1 correct

b6025781

0.0%0/1 correct

a89d7624

100.0%1/1 correct

b0479f84

0.0%0/1 correct

1d4e3b97

100.0%1/1 correct

07b6f563

100.0%1/1 correct

1c0ddc50

100.0%1/1 correct

0a34ad58

100.0%1/1 correct

d3ab962e

100.0%1/1 correct

2311e44b

100.0%1/1 correct

cc06de0d

100.0%1/1 correct

a11281a2

100.0%1/1 correct

4f54b7c9

100.0%1/1 correct

85fa3a3f

100.0%1/1 correct

9aaed6a3

100.0%1/1 correct

1f2b8d4f

100.0%1/1 correct

e6041065

100.0%1/1 correct

51c32626

0.0%0/1 correct

d905b33f

100.0%1/1 correct

7405e8b1

100.0%1/1 correct

f35224e0

100.0%1/1 correct

6456829e

100.0%1/1 correct

a4996e51

100.0%1/1 correct

3c1045c8

100.0%1/1 correct

60036106

100.0%1/1 correct

681a1674

100.0%1/1 correct

e25c3b8d

100.0%1/1 correct

4adc0475

100.0%1/1 correct

4bc144e2

100.0%1/1 correct

ef66a6e5

100.0%1/1 correct

5025383b

100.0%1/1 correct

a1cc6108

100.0%1/1 correct

9ee3ecd6

100.0%1/1 correct

3fdac837

100.0%1/1 correct

91b15a6e

100.0%1/1 correct

27016adc

100.0%1/1 correct

720133ac

100.0%1/1 correct

77eafa52

100.0%1/1 correct

8979f9ec

100.0%1/1 correct

0100672e

100.0%1/1 correct

a96c20ee

100.0%1/1 correct

92a0aa75

100.0%1/1 correct

3fe836c9

100.0%1/1 correct

1c549ce4

100.0%1/1 correct

6c49646a

100.0%1/1 correct

1192316e

100.0%1/1 correct

0ea62687

100.0%1/1 correct

67e0d0f2

100.0%1/1 correct

bb7c3b45

100.0%1/1 correct

ba358f49

100.0%1/1 correct

61f8c8f8

100.0%1/1 correct

60159905

100.0%1/1 correct

ef9cf60a

100.0%1/1 correct

73d42213

100.0%1/1 correct

bc149d6b

100.0%1/1 correct

099778bb

100.0%1/1 correct

09ba9854

0.0%0/1 correct

d6062bb9

100.0%1/1 correct

157a136e

100.0%1/1 correct

c18a7dc8

100.0%1/1 correct

a3332713

100.0%1/1 correct

55241a1f

100.0%1/1 correct

a08a253f

100.0%1/1 correct

f0e564bc

100.0%1/1 correct

078150f1

100.0%1/1 correct

8cf4d046

100.0%1/1 correct

a346bb18

100.0%1/1 correct

37f165cf

0.0%0/1 correct

8e91e7d9

100.0%1/1 correct

87f22b4a

100.0%1/1 correct

e56a43b9

100.0%1/1 correct

efc3f7c2

100.0%1/1 correct

21d02d0d

100.0%1/1 correct

2311e44b_abs

100.0%1/1 correct

6456829e_abs

100.0%1/1 correct

e5ba910e_abs

100.0%1/1 correct

a96c20ee_abs

0.0%0/1 correct

ba358f49_abs

100.0%1/1 correct

09ba9854_abs

0.0%0/1 correct

gpt4_59149c77

100.0%1/1 correct

gpt4_f49edff3

100.0%1/1 correct

71017276

100.0%1/1 correct

b46e15ed

100.0%1/1 correct

gpt4_fa19884c

100.0%1/1 correct

0bc8ad92

100.0%1/1 correct

af082822

100.0%1/1 correct

gpt4_4929293a

100.0%1/1 correct

gpt4_b5700ca9

100.0%1/1 correct

9a707b81

0.0%0/1 correct

gpt4_1d4ab0c9

100.0%1/1 correct

gpt4_e072b769

100.0%1/1 correct

0db4c65d

100.0%1/1 correct

gpt4_1d80365e

100.0%1/1 correct

gpt4_7f6b06db

0.0%0/1 correct

gpt4_6dc9b45b

100.0%1/1 correct

gpt4_8279ba02

100.0%1/1 correct

gpt4_18c2b244

100.0%1/1 correct

gpt4_a1b77f9c

100.0%1/1 correct

gpt4_1916e0ea

100.0%1/1 correct

gpt4_7a0daae1

100.0%1/1 correct

gpt4_468eb063

100.0%1/1 correct

gpt4_7abb270c

0.0%0/1 correct

gpt4_1e4a8aeb

100.0%1/1 correct

gpt4_4fc4f797

100.0%1/1 correct

4dfccbf7

0.0%0/1 correct

gpt4_61e13b3c

100.0%1/1 correct

gpt4_45189cb4

100.0%1/1 correct

2ebe6c90

100.0%1/1 correct

gpt4_e061b84f

0.0%0/1 correct

370a8ff4

0.0%0/1 correct

gpt4_d6585ce8

100.0%1/1 correct

gpt4_4ef30696

100.0%1/1 correct

gpt4_ec93e27f

100.0%1/1 correct

6e984301

100.0%1/1 correct

8077ef71

100.0%1/1 correct

gpt4_f420262c

100.0%1/1 correct

gpt4_8e165409

100.0%1/1 correct

gpt4_74aed68e

100.0%1/1 correct

bcbe585f

100.0%1/1 correct

gpt4_21adecb5

100.0%1/1 correct

5e1b23de

100.0%1/1 correct

gpt4_98f46fc6

100.0%1/1 correct

gpt4_af6db32f

100.0%1/1 correct

eac54adc

100.0%1/1 correct

gpt4_7ddcf75f

100.0%1/1 correct

gpt4_a2d1d1f6

100.0%1/1 correct

gpt4_85da3956

0.0%0/1 correct

gpt4_b0863698

100.0%1/1 correct

gpt4_68e94287

100.0%1/1 correct

gpt4_e414231e

100.0%1/1 correct

gpt4_7ca326fa

100.0%1/1 correct

gpt4_7bc6cf22

100.0%1/1 correct

2ebe6c92

100.0%1/1 correct

gpt4_e061b84g

100.0%1/1 correct

71017277

100.0%1/1 correct

b46e15ee

100.0%1/1 correct

gpt4_d6585ce9

0.0%0/1 correct

gpt4_1e4a8aec

100.0%1/1 correct

gpt4_f420262d

100.0%1/1 correct

gpt4_59149c78

100.0%1/1 correct

gpt4_e414231f

100.0%1/1 correct

gpt4_4929293b

100.0%1/1 correct

gpt4_468eb064

100.0%1/1 correct

gpt4_fa19884d

100.0%1/1 correct

9a707b82

100.0%1/1 correct

eac54add

0.0%0/1 correct

4dfccbf8

100.0%1/1 correct

0bc8ad93

100.0%1/1 correct

6e984302

100.0%1/1 correct

gpt4_8279ba03

100.0%1/1 correct

gpt4_b5700ca0

100.0%1/1 correct

gpt4_68e94288

100.0%1/1 correct

gpt4_2655b836

100.0%1/1 correct

gpt4_2487a7cb

100.0%1/1 correct

gpt4_76048e76

100.0%1/1 correct

gpt4_2312f94c

100.0%1/1 correct

0bb5a684

100.0%1/1 correct

08f4fc43

100.0%1/1 correct

2c63a862

100.0%1/1 correct

gpt4_385a5000

100.0%1/1 correct

2a1811e2

100.0%1/1 correct

bbf86515

100.0%1/1 correct

gpt4_5dcc0aab

100.0%1/1 correct

gpt4_0b2f1d21

100.0%1/1 correct

f0853d11

100.0%1/1 correct

gpt4_6ed717ea

100.0%1/1 correct

gpt4_70e84552

100.0%1/1 correct

a3838d2b

100.0%1/1 correct

gpt4_93159ced

100.0%1/1 correct

gpt4_2d58bcd6

100.0%1/1 correct

gpt4_65aabe59

100.0%1/1 correct

982b5123

100.0%1/1 correct

b9cfe692

100.0%1/1 correct

gpt4_4edbafa2

100.0%1/1 correct

c8090214

100.0%1/1 correct

gpt4_483dd43c

100.0%1/1 correct

e4e14d04

100.0%1/1 correct

c9f37c46

100.0%1/1 correct

gpt4_2c50253f

100.0%1/1 correct

dcfa8644

100.0%1/1 correct

gpt4_b4a80587

100.0%1/1 correct

gpt4_9a159967

100.0%1/1 correct

cc6d1ec1

100.0%1/1 correct

gpt4_8c8961ae

100.0%1/1 correct

gpt4_d9af6064

100.0%1/1 correct

gpt4_7de946e7

100.0%1/1 correct

d01c6aa8

100.0%1/1 correct

993da5e2

100.0%1/1 correct

a3045048

100.0%1/1 correct

gpt4_d31cdae3

100.0%1/1 correct

gpt4_cd90e484

100.0%1/1 correct

gpt4_88806d6e

100.0%1/1 correct

gpt4_4cd9eba1

100.0%1/1 correct

gpt4_93f6379c

100.0%1/1 correct

b29f3365

100.0%1/1 correct

gpt4_2f56ae70

100.0%1/1 correct

6613b389

100.0%1/1 correct

gpt4_78cf46a3

100.0%1/1 correct

gpt4_0a05b494

100.0%1/1 correct

gpt4_1a1dc16d

100.0%1/1 correct

gpt4_2f584639

100.0%1/1 correct

gpt4_213fd887

100.0%1/1 correct

gpt4_5438fa52

100.0%1/1 correct

gpt4_c27434e8

100.0%1/1 correct

gpt4_fe651585

0.0%0/1 correct

8c18457d

100.0%1/1 correct

gpt4_70e84552_abs

0.0%0/1 correct

gpt4_93159ced_abs

0.0%0/1 correct

982b5123_abs

100.0%1/1 correct

c8090214_abs

100.0%1/1 correct

gpt4_c27434e8_abs

100.0%1/1 correct

gpt4_fe651585_abs

100.0%1/1 correct

6a1eabeb

100.0%1/1 correct

6aeb4375

100.0%1/1 correct

830ce83f

100.0%1/1 correct

852ce960

100.0%1/1 correct

945e3d21

100.0%1/1 correct

d7c942c3

100.0%1/1 correct

71315a70

100.0%1/1 correct

89941a93

100.0%1/1 correct

ce6d2d27

100.0%1/1 correct

9ea5eabc

100.0%1/1 correct

07741c44

100.0%1/1 correct

a1eacc2a

100.0%1/1 correct

184da446

100.0%1/1 correct

031748ae

100.0%1/1 correct

4d6b87c8

100.0%1/1 correct

0f05491a

100.0%1/1 correct

08e075c7

100.0%1/1 correct

f9e8c073

100.0%1/1 correct

41698283

100.0%1/1 correct

2698e78f

100.0%1/1 correct

b6019101

100.0%1/1 correct

45dc21b6

100.0%1/1 correct

5a4f22c0

100.0%1/1 correct

6071bd76

100.0%1/1 correct

e493bb7c

100.0%1/1 correct

618f13b2

100.0%1/1 correct

72e3ee87

100.0%1/1 correct

c4ea545c

100.0%1/1 correct

01493427

100.0%1/1 correct

6a27ffc2

100.0%1/1 correct

2133c1b5

100.0%1/1 correct

18bc8abd

100.0%1/1 correct

db467c8c

100.0%1/1 correct

7a87bd0c

100.0%1/1 correct

e61a7584

100.0%1/1 correct

1cea1afa

100.0%1/1 correct

ed4ddc30

100.0%1/1 correct

8fb83627

100.0%1/1 correct

b01defab

100.0%1/1 correct

22d2cb42

100.0%1/1 correct

0e4e4c46

100.0%1/1 correct

4b24c848

100.0%1/1 correct

7e974930

100.0%1/1 correct

603deb26

100.0%1/1 correct

59524333

100.0%1/1 correct

5831f84d

100.0%1/1 correct

eace081b

100.0%1/1 correct

affe2881

100.0%1/1 correct

50635ada

100.0%1/1 correct

e66b632c

100.0%1/1 correct

0ddfec37

100.0%1/1 correct

f685340e

100.0%1/1 correct

cc5ded98

100.0%1/1 correct

dfde3500

100.0%1/1 correct

69fee5aa

100.0%1/1 correct

7401057b

100.0%1/1 correct

cf22b7bf

100.0%1/1 correct

a2f3aa27

0.0%0/1 correct

c7dc5443

100.0%1/1 correct

06db6396

100.0%1/1 correct

3ba21379

100.0%1/1 correct

9bbe84a2

100.0%1/1 correct

10e09553

100.0%1/1 correct

dad224aa

100.0%1/1 correct

ba61f0b9

100.0%1/1 correct

42ec0761

100.0%1/1 correct

5c40ec5b

100.0%1/1 correct

c6853660

100.0%1/1 correct

26bdc477

100.0%1/1 correct

0977f2af

100.0%1/1 correct

6aeb4375_abs

100.0%1/1 correct

031748ae_abs

0.0%0/1 correct

2698e78f_abs

100.0%1/1 correct

2133c1b5_abs

0.0%0/1 correct

0ddfec37_abs

100.0%1/1 correct

f685340e_abs

100.0%1/1 correct

89941a94

100.0%1/1 correct

07741c45

0.0%0/1 correct

7161e7e2

100.0%1/1 correct

c4f10528

100.0%1/1 correct

89527b6b

0.0%0/1 correct

e9327a54

100.0%1/1 correct

4c36ccef

100.0%1/1 correct

6ae235be

100.0%1/1 correct

7e00a6cb

100.0%1/1 correct

1903aded

100.0%1/1 correct

ceb54acb

100.0%1/1 correct

f523d9fe

100.0%1/1 correct

0e5e2d1a

100.0%1/1 correct

fea54f57

100.0%1/1 correct

cc539528

100.0%1/1 correct

dc439ea3

100.0%1/1 correct

18dcd5a5

100.0%1/1 correct

488d3006

100.0%1/1 correct

58470ed2

0.0%0/1 correct

8cf51dda

100.0%1/1 correct

1d4da289

100.0%1/1 correct

8464fc84

100.0%1/1 correct

8aef76bc

100.0%1/1 correct

71a3fd6b

100.0%1/1 correct

2bf43736

100.0%1/1 correct

70b3e69b

100.0%1/1 correct

8752c811

100.0%1/1 correct

3249768e

100.0%1/1 correct

1b9b7252

100.0%1/1 correct

1568498a

100.0%1/1 correct

6222b6eb

100.0%1/1 correct

e8a79c70

100.0%1/1 correct

d596882b

100.0%1/1 correct

e3fc4d6e

100.0%1/1 correct

51b23612

100.0%1/1 correct

3e321797

100.0%1/1 correct

e982271f

100.0%1/1 correct

352ab8bd

100.0%1/1 correct

fca762bc

100.0%1/1 correct

7a8d0b71

100.0%1/1 correct

a40e080f

100.0%1/1 correct

8b9d4367

100.0%1/1 correct

5809eb10

100.0%1/1 correct

41275add

100.0%1/1 correct

4388e9dd

100.0%1/1 correct

4baee567

100.0%1/1 correct

561fabcd

100.0%1/1 correct

b759caee

100.0%1/1 correct

ac031881

100.0%1/1 correct

28bcfaac

100.0%1/1 correct

16c90bf4

100.0%1/1 correct

c8f1aeed

100.0%1/1 correct

eaca4986

100.0%1/1 correct

c7cf7dfd

100.0%1/1 correct

e48988bc

100.0%1/1 correct

1de5cff2

100.0%1/1 correct

65240037

100.0%1/1 correct

778164c6

100.0%1/1 correct