Inquiry for ARC Performance

[AllenPiaoHong]

Thanks for sharing the amazing project.
But I have a question why the performance of GPT-4o in arc dataset can only reach 7% ?

Looking for your reply !

[ShengranHu]

Hi! Thanks for checking out. The naming of this augment is a bit confusing because args.model only controls the model used in the meta agent. If you would like to change the model used for evaluating the generated agents, please change the default value defined here.

[Sanxianxiajiao]

@AllenPiaoHong Hi, have you successfully reproduced the result?

[Babylonehy]

no，i can not