Skip to content

agentSystem.forward is not working in MMLU task #10

@kwangje

Description

@kwangje

I'm having an issue where the content of the results is coming out empty. Here is a snippet of the code and result:

The result for task_queue[99] is:

Info(name='task', author='User', content='Answer the following multiple choice question.\n\nWhen a fusion reaction converts a pair of hydrogen isotopes to an alpha particle and a neutron most of the energy released is in the form of\n\n(A) gamma radiation.\n(B) kinetic energy of the alpha particle.\n(C) kinetic energy of the neutron.\n(D) All of these about equally.', iteration_idx=-1)
agentSystem = AgentSystem()

acc_list = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
    results = list(tqdm(executor.map(agentSystem.forward, task_queue), total=len(task_queue)))

The result for results[0] is:

Info(name='final_answer', author='Self-Critique Agent', content=Info(name='answer', author='Answer Generator y5Cv', content='', iteration_idx=-1), iteration_idx=-1)

The results of the code always show that the content is empty. I am not sure why this is happening.

Could it be that the issue arises because the code implementations for MMLU and other tasks are missing, whereas the AgentSystem class works fine for ARC tasks?

I would appreciate any help you can provide.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions