Incorrect opt_end_learning_rate for GBS 64 in gpt-oss-20b RCPs

[ramgandikota]

While collecting submission logs and comparing against RCPs, we found that the GBS 64 RCPs use an LR of "1e-05" while using a MIN_LR of "4e-05". This would violate the rule for the hyperparameters of this benchmark:
https://github.com/mlcommons/training_policies/blob/master/training_rules.adoc#91-hyperparameters

gpt_oss_20b | adamw | opt_end_learning_rate | opt_base_learning_rate * 0.1 

Logs from reference RCPs:
https://github.com/mlcommons/training/blob/master/small_llm_moe_pretraining/primus/rcp_logs/gbs64/run_0.log#L30

Logs snippet:

:::MLLOG {"namespace": "", "time_ms": 1773021880374, "event_type": "POINT_IN_TIME", "key": "opt_adamw_epsilon", "value": 1e-05, "metadata": {"file": "/opt/venv/lib/python3.10/site-packages/primus_mllog/mlperf_pre_training.py", "lineno": 65}}
:::MLLOG {"namespace": "", "time_ms": 1773021880374, "event_type": "POINT_IN_TIME", "key": "opt_adamw_weight_decay", "value": 0.1, "metadata": {"file": "/opt/venv/lib/python3.10/site-packages/primus_mllog/mlperf_pre_training.py", "lineno": 65}}
:::MLLOG {"namespace": "", "time_ms": 1773021880374, "event_type": "POINT_IN_TIME", "key": "opt_gradient_clip_norm", "value": 1.0, "metadata": {"file": "/opt/venv/lib/python3.10/site-packages/primus_mllog/mlperf_pre_training.py", "lineno": 65}}
:::MLLOG {"namespace": "", "time_ms": 1773021880374, "event_type": "POINT_IN_TIME", "key": opt_end_learning_rate", "value": 4e-05, "metadata": {"file": "/opt/venv/lib/python3.10/site-packages/primus_mllog/mlperf_pre_training.py", "lineno": 65}}

[ShriyaRishab]

This is indeed an oversight in the GBS 64 RCP for GPT-OSS-20B. We suspect that it slipped through the cracks during review because the check for opt_end_learning_rate is not implemented in the compliance checker.

Given that we are so close to the submission and generation of new RCPs would take AMD ~80hrs, deferring this to be fixed after v6.0. For now, using MinLR of 4e-5 even though it is not equal to 1e-5 * 0.1 will be permitted for GPT-OSS-20B GBS 64.

[ShriyaRishab]

Further analysis shows that all the RCPs are using opt_end_learning_rate of 4e-5 irrespective of the LR because of this line in the yaml config.

 lr: ${PRIMUS_LR:4.0e-4}  # Reduced from 8e-4 for FP8 stability
 min_lr: ${PRIMUS_MIN_LR:4.0e-5}  # Set to 10% of max LR

Unfortunately, if only PRIMUS_LR is set and PRIMUS_MIN_LR is not set, PRIMUS_MIN_LR defaults to 4e-5 and that's what has happened in all the RCPs. We don't expect convergence to change much due to PRIMUS_MIN_LR but since we're very close to v6.0 submission, only for GPT-OSS-20B using opt_end_learning_rate = 4e-5 or opt_end_learning_rate = opt_base_learning_rate * 0.1 should be accepted as long as they pass the RCPs.