[RLVMR] Success rate definition in code does not match paper description

[luli-git]

Hi authors, thank you very much for the very insightful work and for releasing your code! While replicating the results of RLVMR on scienceworld, I found that the success rate metric as implemented in the code appears to differ significantly from the definition stated in the paper. The paper defines success rate as "The percentage of tasks successfully completed by the agent on each evaluation split," but the code uses a trivially satisfiable condition that inflates this metric.

Code in Question

In envs.py, the won flag is defined as:

isCompleted = done
info["won"] = isCompleted and info["score"] > 0

And reward is computed as:

def compute_reward(info, multi_modal=False):
    reward = 10.0 * float(info['won'])
    return reward

The Problem

In ScienceWorld, done=True is triggered by three conditions:

1. The agent completes the task (score = 100)
2. The agent hits the step limit (score can be anything from 0 to 99)
3. The agent fails catastrophically (score < 0, forced termination by ScienceWorld)

The > 0 threshold correctly filters out case 3 (catastrophic failures with negative scores). However, it still counts case 2 (step-limit timeouts) as successes, even when the agent made zero progress (score = 0).

Empirical Evidence

When training with this code, I observed:

Success rate (won) converged to ~100% — which is expected since every finished episode counts as a "win"
Meanwhile, the actual score (logged when won=True) decreased to ~2.1 (out of 100), indicating the agent was not meaningfully completing tasks. This confirms that the success rate metric is not measuring task completion.

Expected Behavior

A meaningful success rate definition should require actual task completion, e.g.:

info["won"] = isCompleted and info["score"] >= 100 # fully completed

Questions

1. Is this an intentional design choice, or a bug (e.g., should > 0 be >= 100)?
2. Were the results reported in the paper evaluated using this same code, or was a different success criterion used during evaluation?

[LGXNBXN]

I also encountered the same issue, but my problem only occurs during the L2 level training on SciWorld.

[zzjweb]

Hi, thanks for your feedback. In our experiments, the step limit in the ScienceWorld environment is set to a very large value, and LLM-agent-controlled interactions typically never hit this cap. So in practice we only encounter cases (1) and (3). We filter out catastrophic failures by requiring done and info["score"] > 0.

Based on our prior experience, if you observe the success rate converging to ~100%, that seems anomalous. You might want to print/log a subset of trajectories to check whether any completed-but-catastrophic runs are being incorrectly assigned positive feedback, or try using the cumulative score as the feedback signal instead.

[luli-git]

Hi, thanks for your feedback. In our experiments, the step limit in the ScienceWorld environment is set to a very large value, and LLM-agent-controlled interactions typically never hit this cap. So in practice we only encounter cases (1) and (3). We filter out catastrophic failures by requiring done and info["score"] > 0.

Based on our prior experience, if you observe the success rate converging to ~100%, that seems anomalous. You might want to print/log a subset of trajectories to check whether any completed-but-catastrophic runs are being incorrectly assigned positive feedback, or try using the cumulative score as the feedback signal instead.

Thank you very much for your reply! Just to confirm, is the step limit you set for the ScienceWorld env to be 30 (https://github.com/Tencent/digitalhuman/blob/main/RLVMR/code/examples/gigpo_trainer/run_sciworld.sh#L56C4-L56C19)?

[Jerrrrykun]

Hi, thanks for your feedback. In our experiments, the step limit in the ScienceWorld environment is set to a very large value, and LLM-agent-controlled interactions typically never hit this cap. So in practice we only encounter cases (1) and (3). We filter out catastrophic failures by requiring done and info["score"] > 0.
Based on our prior experience, if you observe the success rate converging to ~100%, that seems anomalous. You might want to print/log a subset of trajectories to check whether any completed-but-catastrophic runs are being incorrectly assigned positive feedback, or try using the cumulative score as the feedback signal instead.

Thank you very much for your reply! Just to confirm, is the step limit you set for the ScienceWorld env to be 30 (https://github.com/Tencent/digitalhuman/blob/main/RLVMR/code/examples/gigpo_trainer/run_sciworld.sh#L56C4-L56C19)?

I think what the authors meant: the env max step is 100 for Sciworld, and in the training script, the LLM's max step is 30, which means it would collect the trajs, no matter if it's finished or not. In this case, most of the partially rewarded trajs' isCompleted=False. So we only have the case (1) and case (3).

But if my above understanding is correct, may I ask @zzjweb why you said this? Since we only have case (1) and (3), why does the success rate converge to 100% is abnormal?

Based on our prior experience, if you observe the success rate converging to ~100%, that seems anomalous. You might want to print/log a subset of trajectories to check whether any completed-but-catastrophic runs are being incorrectly assigned positive feedback, or try using the cumulative score as the feedback signal instead.