Questions about data preparation for DPO

[hyeonji21]

There is no code provided for the DPO execution process described in the paper after RM scoring.
Can you explain how you built the data for DPO?

[Linear95]

Sorry for replying late. After RM scoring, we use the response with the highest score as the positive sample for DPO and the one with the lowest as the negative sample. For RRHF, all four responses are used. For rejection sampling, only the response with the highest score is utilized.