Bootstrap-only rule deletion leaves stale completion annotations, causing evaluation bypass on rule recreation

[Karman580]

What happened

When a NodeReadinessRule using enforcementMode: bootstrap-only is deleted, the controller removes the taints managed by the rule, but the bootstrap completion annotations written on nodes are left behind.

The annotation key is generated using the rule name:

go annotationKey := fmt.Sprintf("readiness.k8s.io/bootstrap-completed-%s", ruleName) 

isBootstrapCompleted() later checks this annotation to decide whether node evaluation should be skipped.

Because the annotation only depends on the rule name, recreating a rule with the same name causes previously-bootstrapped nodes to be treated as already completed, even if the new rule has different conditions.

As a result, node evaluation is skipped and the new taint may never be applied.

I traced this to the delete path in reconcileDelete(). It currently cleans up taints through cleanupTaintsForRule() but does not clean up bootstrap annotations before removing the finalizer.

What you expected to happen

Deleting a bootstrap-only rule should also remove the bootstrap completion state associated with that rule.

If a rule is recreated with the same name, nodes should be evaluated from scratch.

How to reproduce

# Create bootstrap-only rule
kubectl apply -f - <<EOF
apiVersion: readiness.node.x-k8s.io/v1alpha1
kind: NodeReadinessRule
metadata:
  name: network-readiness
spec:
  enforcementMode: bootstrap-only
  conditions:
  - type: NetworkUnavailable
    requiredStatus: "False"
  taint:
    key: readiness.k8s.io/network-not-ready
    effect: NoSchedule
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
EOF

# Wait for bootstrap completion

# Delete the rule
kubectl delete nodereadinessrule network-readiness

# Verify annotations still exist
kubectl get nodes -o json | \
jq '.items[] | select(
  .metadata.annotations["readiness.k8s.io/bootstrap-completed-network-readiness"] != null
) | .metadata.name'

# Recreate rule with same name but different condition
kubectl apply -f - <<EOF
apiVersion: readiness.node.x-k8s.io/v1alpha1
kind: NodeReadinessRule
metadata:
  name: network-readiness
spec:
  enforcementMode: bootstrap-only
  conditions:
  - type: DiskPressure
    requiredStatus: "False"
  taint:
    key: readiness.k8s.io/network-not-ready
    effect: NoSchedule
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
EOF

At this point, nodes that previously completed bootstrap can skip evaluation because the old annotation is still present.

Suggested fix

A few possible directions:

• Remove bootstrap annotations during finalizer cleanup
• Use rule UID instead of rule name in the annotation key
• Combine UID-based keys with best-effort cleanup during deletion

Happy to work on a fix if this approach makes sense.

/kind bug

[Karman580]

/assign

[Priyankasaggu11929]

same as I suggested on other PR - #253 (comment)

Thank you. :)

[Karman580]

Thanks for the feedback. I reran the reproduction locally and rewrote the gist with actual kind setup, manifests, controller logs, kubectl outputs, annotation/taint state, and findings from local testing only.

Updated gist:
https://gist.github.com/Karman580/edd3e3ae032ba08b66a4555e266144d7

[ajaysundark]

Bootstrap-only rule deletion leaves stale completion annotations, causing evaluation bypass on rule recreation

I dont think if this to be treated a bug. We dont have user data to decide, would it be to expected to support a scenario like this in real clusters.
I could imagine if the original conditions were determined needing an update (we dont allow mutating rule spec), user may want to delete and recreate a rule.
But would we want to clean up the annotations from all the node for it, or determine node reevaluation based on right node state.

We can evaluate this use-case for a "rule recreation protection" via UIDs in the rule-status, when we are looking for a better design for our annotation.
#224 (comment)

Let's treat this as a backlog.

kind/backlog

[Karman580]

Thanks for the clarification and the detailed design context.

That makes sense especially around whether rule recreation should be treated as a fresh evaluation boundary or preserved bootstrap state. I agree this likely fits better as a broader annotation/rule-state design discussion rather than just isolated cleanup logic.

Appreciate the feedback. I’ll keep following the annotation design discussion in #224 as well.