Questions about reproducing training results with default configuration

[li-tianqi]

Reproduction Environment

• GPU: H20*4
• Configuration: Codebase default settings
• Dataset: First 15k SAT samples (as per default config)

My Results

Figure 1. My training curve

Figure 2. My test curve

• My reproduction (base+GRPO): 58.4 (step 1000)
• Qwen2-VL instruct model: 61.6

Results from report

Figure 3. Test curve from report

Key Questions

1. Performance gap (58.4 vs ~59.5) between my reproduction and reported results.
2. Inconsistent qwen2-VL instruct model performance (61.6 locally vs ~56 in report).
3. Abnormal trend in reproduced SFT curve and GRPO curve (Figure 2) .
4. Why does default config only use first 15k SAT samples instead of full dataset?
5. According to your experience, what are the possible reasons for the abnormal reproduction results?

[caihong06302923]

Same phenomenon

[ruolinsss]

same training curve with different training dataset

[HengguangZhou]

Thank you for sharing your results and for bringing these important points to our attention. Below are our responses to your questions:

1. Performance Gap (58.4 vs ~59.5):
   • While this difference exists, we believe it falls within the expected range of variance in RL training. Please consider rerunning the training to check for variance. For reference, we have released our model checkpoint at step 1500 on Hugging Face.
2. Qwen2-VL Instruct Model Performance:
   • Thank you for pointing this out — this was indeed a oversight on our end. We will update our reports to correct this. However, please note that the Qwen2-VL Instruct model was fine-tuned on significantly more data, so comparing it directly with our method isn't entirely fair.
3. Abnormal SFT Curve and GRPO curve:
   • I can not reproduce the SFT evaluation you mentioned on my end, but the abnormal trend observed in the reproduced SFT curve may be due to a bug in the evaluation process. Since the chat_template.json file is not saved in SFTTrainer when performing SFT on the base model, this issue arises. This [fix](c7aad5c) introduces a workaround that downloads the chat_template.json from Hugging Face to resolve the problem.
   • The trend observed in the reproduced GRPO curve is likely because default configuration limits the maximum completion length to 700 tokens. Any response longer than 700 token will have its final answer truncated, and the model gets 0 reward for longer response. As a result, the model eventually decrease its response length.
4. Why Only the First 15k SAT Samples:
   • We trained for 18k steps, and the response length and performance curves remained consistent after 15k steps, so we stopped training further. We did not fully use the 15k SAT samples neither, instead we only utilized the subset corresponding to the first 1.5k steps (1.5k*4 = 6 k samples).
5. Abnormal Reproduction Results:
   • For SFT, please try the latest fix we just pushed to the SFT script, it may resolve the issue. For the GRPO curve, consider increasing the response length setting. And please note that RL training can be quite unstable and subject to high variance, so it’s worth running it a few times to check.

Please feel free to reach out if you have further questions or would like us to expand on any specific points. We value your feedback and are committed to improving our work.