Enquiry about reproducing results on ARC

[dercaft]

Hi, I have three questions:

1/3 Evaluation of searched code:

I use the searched code in results/arc_gpt3.5_results.jsonl, which has Median: 13.7%, Line 147.
But I get a much lower result, 95% Bootstrap Confidence Interval: (1.3%, 5.7%), Median: 3.3%.

2/3 Running search.py and evaluation:

python search.py

The best result is "95% Bootstrap Confidence Interval: (3.0%, 14.0%), Median: 8.0%", which still has gap to results in paper.

3/3 Results based on sampled training dataset, but not from evaluation dataset?

Looking forward to your reply.

[ShengranHu]

Thank you for checking out our work!

We reported the performance on the evaluation set in the paper. For transparency, both the results for the training and evaluation datasets are included in the experiment log.

There can definitely be variance in individual runs of our algorithm due to stochastic LLM sampling, especially in the ARC challenge. In fact, when I attempted to evaluate the agent you're referring to, the result was lower than the run reported in the paper, but it fell within our expected error bounds (see below figure).

Given this variability, we would recommend running multiple evaluations to get a more comprehensive view of the performance range. However, we noticed that your results seem to be significantly outside our reported error bounds. In this case, it might be helpful to double-check if all settings and models are correctly configured.

Please don't hesitate to reach out if you have any more questions or if there's anything else we can clarify.

[Sanxianxiajiao]

Hi, I would like to add your WeChat or QQ to discuss in detail. Is that ok? :D @ShengranHu

[ShengranHu]

Hello @Sanxianxiajiao. Please feel free to post your follow-up questions here or contact me via my email :)

[Sanxianxiajiao]

Hello @Sanxianxiajiao. Please feel free to post your follow-up questions here or contact me via my email :)

I have sent you an email, please check your mailbox when you have time. ：D