Fix to prevent consecutive reboots in case of reboot delay.

[hlts2]

WHAT

This PR contains the following changes:

• Implemented the isLastRebootCmdTimeAfter function, which checks if the last reboot command time for a specified node is after the given threshold time.

WHY

In this case, there was a scenario where the reboot was delayed for over 60 minutes. As a result, the node's LastTransitionTime remained unchanged, causing reboot commands to be issued every time. In other words, since the node's LastTransitionTime didn't change for 60 minutes, the node-agent's check loop continued to trigger, resulting in multiple reboot commands being sent during that time.

To address this, I considered it necessary to use both the node's LastTransitionTime and LastRebootCmdTime (the actual time the reboot command was sent) to handle such cases. This dual comparison ensures that reboot commands are not repeatedly triggered in the event of delays, effectively preventing unnecessary reboots.

[jokestax]

Hey @hlts2 ,if we are storing the LastRebootCmdTime,i think we can remove LastTransitionTime

[hlts2]

@jokestax

Hi, thank you for your comments. I believe both LastRebootCmdTime and LastTransitionTime are necessary.

For example, in the following case, handling it with just LastRebootCmdTime would not be sufficient:

Condition:

• LastRebootCmdTime: 2 hours ago (from the current time)
• Time window: 1 hour

K8s nodes can temporarily become NotReady due to network issues or other factors, meaning they may transition between NotReady → Ready → NotReady. In such cases, relying only on LastRebootCmdTime might cause a reboot command to be issued immediately.

What really matters is how long the node has been in the NotReady state. To track this, LastTransitionTime is necessary.

Additionally, after a reboot command is issued, the reboot might be delayed for various reasons. To prevent the same reboot command from being triggered again, it is important to track when the reboot command was last sent. This is where LastRebootCmdTime comes into play, as it prevents the command from being reissued before a certain amount of time has passed.

In summary, we use LastTransitionTime to track how long a node has been in the NotReady state, and LastRebootCmdTime to prevent the same reboot command from being triggered multiple times, especially if the reboot is delayed. So I think this approach enhances the existing logic. What do you think? 🤔

[jokestax]

i got it,below i have listed the cases to make it more clear

 let window = 60 min
 
- LTT > 60 , LRCT < 60 dont reboot
- LTT < 60 , LRCT < 60 dont reboot
- LTT < 60 , LRCT > 60 dont reboot
- LTT > 60, LRCT >. 60 reboot

[hlts2]

@jokestax

i got it,below i have listed the cases to make it more clear

OK, Thank you for your comment. 🙏
I have fixed it with the following commit.

e34999a

[hlts2]

@jokestax Thank you for your review 🙇 I will merge this PR 🚀