Validate version skew policy compliance on cluster upgrades by kke · Pull Request #923 · k0sproject/k0sctl

kke · 2025-08-12T11:26:16Z

When upgrading, validate that the new version does not violate kube's version skew policy or upgrade path requirements.

Waits for kube-proxy new version roll-out after controller upgrades before starting worker upgrades.

jnummelin · 2025-08-12T11:48:18Z

+		return nil
+	}
+
+	if !delta.Consecutive {


What does Consecutive mean here, X+1 minor version?

1.29.3 => 1.30.0 ✅
1.29.3 => 1.31.0 ❌
1.29.3 => 2.0.0 ✅
1.29.3 => 2.1.0 ❌

...but

1.29.3 => 1.30.1 ❌ (this is probably too strict and should be allowed)

Allowing jumps of minor+2 (1.29 => 1.31) would be ok by kube I suppose.

Confused 🤷‍♂️

kubeadm docs state explicitly that upgrading more than one minor at a time is not supported.

It seems it's kube-apiserver, controller manager and schedular that do not support upgrades that skip minors.

In theory, workers could lag 3 minors behind but I don't see how that would happen with k0sctl unless you add previously unmanaged nodes and proceed to upgrade from 1.28 to 1.31.

Major upgrades are not supported at all, but I guess that is something for the far future.

Essentially it boils down to what k0s supports.

kke · 2025-08-21T12:02:34Z

It could make sense to wait for workers to get the new kube-proxy after control plane is upgraded before proceeding to upgrade them.

Signed-off-by: Kimmo Lehto <klehto@mirantis.com>

Copilot

Pull request overview

Adds upgrade-time validation and sequencing to prevent unsupported Kubernetes/k0s version leaps and to ensure kube-proxy has rolled out before worker upgrades begin.

Changes:

Add version-skew validation during ValidateFacts, blocking unsupported upgrade jumps unless --force is used.
Add a kube-proxy DaemonSet rollout wait gate before starting worker upgrades.
Update smoke tests to exercise the new “blocked without --force” behavior and to apply --force where needed.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
smoke-test/smoke-upgrade.sh	Expects upgrade to fail without `--force`, then succeed with `--force`.
smoke-test/smoke-dryrun.sh	Adjusts apply helper to accept multiple flags and uses `--force` for upgrade paths.
pkg/node/statusfunc.go	Adds DaemonSet rollout verification helpers and kube-proxy-specific rollout check.
phase/validate_facts.go	Introduces version-skew validation for hosts requiring upgrade, with `--force` bypass.
phase/upgrade_workers.go	Waits for kube-proxy rollout (unless `--no-wait`) before upgrading workers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-09T08:39:15Z

+if ../k0sctl apply --config "${K0SCTL_CONFIG}" --debug; then
+  echo "Expected failure when applying without --force"
+  exit 1
+fi
+


The test currently accepts any failure from the first k0sctl apply as an "expected" version-skew rejection. That can hide unrelated regressions (e.g., config or connectivity failures) because it doesn't assert the error cause. Consider capturing the output and checking it contains the version-skew policy error text before treating the failure as expected.

Suggested change

if ../k0sctl apply --config "${K0SCTL_CONFIG}" --debug; then

echo "Expected failure when applying without --force"

exit 1

fi

upgrade_output_file="$(mktemp)"

if ../k0sctl apply --config "${K0SCTL_CONFIG}" --debug >"${upgrade_output_file}" 2>&1; then

echo "Expected failure when applying without --force"

rm -f "${upgrade_output_file}"

exit 1

fi

# Verify that the failure was due to version skew policy

if ! grep -qi "version skew" "${upgrade_output_file}"; then

echo "k0sctl apply failed for an unexpected reason; expected version skew error. Output:"

cat "${upgrade_output_file}"

rm -f "${upgrade_output_file}"

exit 1

fi

rm -f "${upgrade_output_file}"

Copilot · 2026-03-09T08:39:16Z

+	if err := p.validateVersionSkew(); err != nil {
+		return err
+	}


validateVersionSkew adds new upgrade-blocking behavior, but phase/validate_facts_test.go currently only covers downgrade/defaulting/nodeLocalLoadBalancing. Please add unit tests for the new skew rules (controller >1 minor, worker >3 minor, and --force bypass) to prevent regressions.

Copilot · 2026-03-09T08:39:16Z

+    // Upgrade worker hosts parallelly in 10% chunks
+    concurrentUpgrades := int(math.Floor(float64(len(p.hosts)) * float64(p.Config.Spec.Options.Concurrency.WorkerDisruptionPercent/100)))
+    if concurrentUpgrades == 0 {
+        concurrentUpgrades = 1
+    }
+    concurrentUpgrades = min(concurrentUpgrades, p.Config.Spec.Options.Concurrency.Limit)
+
+    // Wait once for kube-proxy to be at desired version across the cluster.
+    if !p.IsWet() {
+        p.DryMsg(p.leader, "wait for kube-proxy to be at the desired version (cluster-wide)")
+    } else if !NoWait { // honor --no-wait
+        log.Infof("waiting for kube-proxy cluster-wide roll-out")
+        if err := retry.WithDefaultTimeout(ctx, node.KubeProxyRolledOutFunc(p.leader)); err != nil {
+            return fmt.Errorf("kube-proxy did not reach the desired version: %w", err)
+        }
+    }

-	log.Infof("Upgrading max %d workers in parallel", concurrentUpgrades)
-	return p.hosts.BatchedParallelEach(ctx, concurrentUpgrades,
-		p.start,
-		p.cordonWorker,
-		p.drainWorker,
-		p.upgradeWorker,
-		p.uncordonWorker,
-		p.finish,
-	)
+    log.Infof("Upgrading max %d workers in parallel", concurrentUpgrades)
+    return p.hosts.BatchedParallelEach(ctx, concurrentUpgrades,
+        p.start,
+        p.cordonWorker,
+        p.drainWorker,
+        p.upgradeWorker,
+        p.uncordonWorker,
+        p.finish,
+    )


This block is indented with spaces instead of the gofmt-standard tabs used throughout the rest of the file. Please run gofmt (or adjust indentation) to keep formatting consistent and avoid noisy diffs/CI style failures.

Copilot · 2026-03-09T08:39:16Z

+func buildLabelSelector(labels map[string]string) string {
+	// Simple AND of matchLabels (k=v,k2=v2,...)
+	if len(labels) == 0 {
+		return ""


DaemonSetRolledOutFunc only builds a selector from spec.selector.matchLabels. If matchLabels is empty/nil (e.g., the DaemonSet uses matchExpressions), buildLabelSelector returns an empty string and the kubectl call ends up listing all pods in the namespace, which can lead to incorrect rollout results/timeouts. Consider either supporting matchExpressions in the JSON struct/selector builder, or returning an explicit error when matchLabels is empty.

Suggested change

return ""

// Returning an empty selector would cause kubectl to list all pods in the namespace,

// which is incorrect if the DaemonSet uses only matchExpressions. Instead, return a

// selector that matches no pods to avoid false rollout success.

log.Warn("buildLabelSelector: empty matchLabels; using a selector that matches no pods (likely matchExpressions-only selector)")

return "k0sctl-selector=no-matching-labels"

kke · 2026-03-09T10:18:06Z

Waiting for k0sproject/k0s#7173

twz123 · 2026-03-11T14:56:19Z

Waiting for k0sproject/k0s#7173

I don't expect this to land. Looks like fire-and-forget AI slop 🤷

kke added the enhancement New feature or request label Aug 12, 2025

jnummelin reviewed Aug 12, 2025

View reviewed changes

kke mentioned this pull request Aug 20, 2025

Document version skew policy k0sproject/k0s#6305

Open

kke changed the title ~~Validate version consicutiveness on upgrade~~ Validate version consecutiveness on upgrade Aug 22, 2025

kke changed the title ~~Validate version consecutiveness on upgrade~~ Validate version skew policy compliance on cluster upgrades Aug 29, 2025

kke force-pushed the version-skew-validation branch from ec260b7 to ccd0855 Compare September 3, 2025 10:12

kke force-pushed the version-skew-validation branch from ccd0855 to 5c47051 Compare September 11, 2025 08:19

kke mentioned this pull request Oct 1, 2025

Available k0s versions for upgrade ? #945

Open

kke force-pushed the version-skew-validation branch from 93a7c0f to 99c7eda Compare March 9, 2026 08:30

kke requested a review from Copilot March 9, 2026 08:31

Copilot started reviewing on behalf of kke March 9, 2026 08:31 View session

Validate version consicutiveness on upgrade

5ae5df1

Signed-off-by: Kimmo Lehto <klehto@mirantis.com>

kke force-pushed the version-skew-validation branch from 99c7eda to 5ae5df1 Compare March 9, 2026 08:35

Copilot AI reviewed Mar 9, 2026

View reviewed changes

-if ../k0sctl apply --config "${K0SCTL_CONFIG}" --debug; then
-  echo "Expected failure when applying without --force"
-  exit 1
-fi
+upgrade_output_file="$(mktemp)"
+if ../k0sctl apply --config "${K0SCTL_CONFIG}" --debug >"${upgrade_output_file}" 2>&1; then
+  echo "Expected failure when applying without --force"
+  rm -f "${upgrade_output_file}"
+  exit 1
+fi
+# Verify that the failure was due to version skew policy
+if ! grep -qi "version skew" "${upgrade_output_file}"; then
+  echo "k0sctl apply failed for an unexpected reason; expected version skew error. Output:"
+  cat "${upgrade_output_file}"
+  rm -f "${upgrade_output_file}"
+  exit 1
+fi
+rm -f "${upgrade_output_file}"

-		return ""
+		// Returning an empty selector would cause kubectl to list all pods in the namespace,
+		// which is incorrect if the DaemonSet uses only matchExpressions. Instead, return a
+		// selector that matches no pods to avoid false rollout success.
+		log.Warn("buildLabelSelector: empty matchLabels; using a selector that matches no pods (likely matchExpressions-only selector)")
+		return "k0sctl-selector=no-matching-labels"

Conversation

kke commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnummelin Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

kke Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

kke Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

kke Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

kke Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

kke commented Aug 21, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

kke commented Mar 9, 2026

Uh oh!

twz123 commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kke commented Aug 12, 2025 •

edited

Loading