[Feature] Fleet: Skip stopped clusters during multi-cluster update in Azure Fleet

[AbhiramVijayan]

Is your feature request related to a problem? Please describe.
We are using Azure Fleet to manage updates across multiple AKS clusters (currently more than 17 clusters).

For cost optimization, some clusters are occasionally placed in a stopped state, but they still remain part of the Fleet configuration.

When running a multi-cluster update strategy, the entire update run fails if even one cluster is in a stopped state. Because of this, the update process stops and none of the other active clusters get updated.

This makes the update process fragile and requires manual intervention before running updates.

Describe the solution you'd like
During a multi-cluster update run, if Fleet detects that a cluster is in a stopped state, it should skip that cluster and continue the update for the remaining active clusters.

This would ensure:

Updates proceed for running clusters

Stopped clusters do not block the entire update

The process becomes more resilient and automated

Describe alternatives you've considered
Describe alternatives you've considered

Currently, the workaround is to:

Manually remove stopped clusters from the update strategy, or

Ensure all clusters are running before starting the update.

However, this is not practical in environments with many clusters and frequent update runs.

Additional context
Example scenario:

Fleet contains 17 clusters

2 clusters are stopped

15 clusters are running

Current behavior:

The update run fails because of the stopped clusters

Expected behavior:

Fleet should skip the stopped clusters and continue updating the remaining 15 running clusters

[sjwaight]

Hi @AbhiramVijayan,

Thanks for the feedback - a really good call out. Handling stopped clusters is certainly something we can improve.

I have a couple of suggested workarounds for you to consider in the interim.

1. Skip clusters: you can already pre-skip clusters in an Update Run. In any Update Run (manual or automated) you can open the Run and find the stopped clusters and mark them as 'skipped'. If the Update Run has already hit a stopped cluster, mark it as skipped (and any other stopped cluster), then re-start the Run. It will run to completion.
2. Failure thresholds (see [Feature] Fleet update runs continue on failure based on setting of acceptable #/% cluster failures #5338): this is currently under development and will allow you to specify a number/% of clusters that can fail on any scope and not block progress of the update. While a stopped cluster shouldn't necessarily be considered a 'failure', this feature would allow you to achieve your outcome without needing to mark clusters as skipped.

[AbhiramVijayan]

Hi @sjwaight thanks for the response and the suggestions.

I tried the first workaround (skipping clusters in the Update Run), and it works as expected. After marking the stopped clusters as skipped, the update continued successfully for the remaining clusters.

However, in environments with a larger number of clusters, this still requires manual intervention during the run. It would be very helpful if Fleet could automatically detect and skip stopped clusters to make the process more seamless.

Also looking forward to the failure thresholds feature you mentioned, as that could help address similar scenarios.

[serbrech]

@AbhiramVijayan thanks for reaching out. We have that on our radar and discussed a few options in the past.
We've kept the default behavior as failure so far, keeping in mind that the goal of Fleet and Updateruns are to keep your fleet uniform.

• automatically skipping stopped clusters would work, but would silently keep some clusters out of date
• auto-start the stopped clusters is controversial, as it could start processes that should not be started in the customer's environment, and could incur non-trivial costs, or unexpected load on shared infrastructure