Skip to content

fix: pre-populate rule cache on first NodeReconciler pass to prevent bootstrap-only taint bypass#254

Open
Karman580 wants to merge 1 commit into
kubernetes-sigs:mainfrom
Karman580:fix/rule-cache-cold-start-race
Open

fix: pre-populate rule cache on first NodeReconciler pass to prevent bootstrap-only taint bypass#254
Karman580 wants to merge 1 commit into
kubernetes-sigs:mainfrom
Karman580:fix/rule-cache-cold-start-race

Conversation

@Karman580
Copy link
Copy Markdown

@Karman580 Karman580 commented May 14, 2026

Description

RuleReadinessController maintains an in-memory ruleCache populated lazily as RuleReconciler processes each NodeReadinessRule. NodeReconciler derives applicable rules exclusively from this cache. Both controllers start consuming their work queues concurrently after informer sync, creating a race where node events are processed against an empty or partially-warm cache.

For continuous enforcement this window is self-healing — the next condition, taint, or label change re-triggers evaluation and the controller converges. For bootstrap-only rules it is a permanent correctness gap: if a node is evaluated before its applicable rule reaches the cache, the taint is never applied, markBootstrapCompleted is never reached, the bootstrap completion annotation is never written, and the intended admission gate is silently bypassed with no error surfaced.

Root Cause

ensureCacheWarmed did not exist. The ruleCache was built entirely as a side effect of RuleReconciler.Reconcile(). Although the controller-runtime manager guarantees informers are synced before controllers start, there was no mechanism to seed ruleCache from the already-available informer data before NodeReconciler began processing events.

Fix

Add ensureCacheWarmed to RuleReadinessController. On the first call it lists all NodeReadinessRules from the already-synced informer cache (no live API round-trip) and adds any rule not yet present in ruleCache. Subsequent calls are no-ops guarded by an atomic.Bool. Existing cache entries — including rules already marked for deletion by RuleReconciler — are intentionally preserved so that in-flight state is not overwritten.

ensureCacheWarmed is called at the top of NodeReconciler.Reconcile before any rule evaluation takes place.

Related Issue

Fixes #253

Type of Change

/kind bug

Testing

  • Added cold-start rule cache warm-up integration test context in node_controller_test.go with two cases:
    • Verifies that a bootstrap-only taint is applied even when ruleCache is empty at reconcile time (the rule is discovered via warm-up).
    • Verifies that the warm-up is not repeated on subsequent reconciles (cacheWarmed remains true).
  • All 49 existing tests continue to pass.
  • Coverage: internal/controller 73.5% → 73.9%.

Checklist

  • make test passes
  • make lint passes

Does this PR introduce a user-facing change?

Fix bootstrap-only taint enforcement correctness gap: NodeReconciler now pre-populates the rule cache from the informer cache on its first reconcile, preventing nodes from bypassing the admission gate during the controller startup race window.

@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label May 14, 2026
@netlify
Copy link
Copy Markdown

netlify Bot commented May 14, 2026

Deploy Preview for node-readiness-controller canceled.

Name Link
🔨 Latest commit 5c83d8f
🔍 Latest deploy log https://app.netlify.com/projects/node-readiness-controller/deploys/6a05d69674a8410008bd86c5

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Karman580
Once this PR has been reviewed and has the lgtm label, please assign ajaysundark for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 14, 2026
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

Hi @Karman580. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@linux-foundation-easycla
Copy link
Copy Markdown

linux-foundation-easycla Bot commented May 14, 2026

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: Karman580 / name: KARMAN SINGH TALWAR (5c83d8f)

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 14, 2026
…bootstrap-only taint bypass

RuleReadinessController maintains an in-memory ruleCache populated lazily
as RuleReconciler processes each NodeReadinessRule. NodeReconciler derives
applicable rules exclusively from this cache. Both controllers start
consuming their work queues concurrently after informer sync, creating a
race where node events are processed against an empty or partially-warm
cache.

For continuous enforcement this window is self-healing. For bootstrap-only
rules it is permanent: if a node is evaluated before its applicable rule
reaches the cache the taint is never applied, the bootstrap completion
annotation is never written, and the admission gate is silently bypassed.

Add ensureCacheWarmed to RuleReadinessController. On the first call it
lists all NodeReadinessRules from the already-synced informer cache and
adds any rule not yet present in ruleCache. Subsequent calls are no-ops
guarded by an atomic.Bool. Existing cache entries (including rules already
marked for deletion by RuleReconciler) are intentionally preserved so that
in-flight state is not overwritten.

Call ensureCacheWarmed at the top of NodeReconciler.Reconcile before any
rule evaluation takes place.

Add two integration tests covering the cold-start scenario:
- verifies that a bootstrap-only taint is applied even when ruleCache is
  empty at reconcile time (rule discovered via warm-up)
- verifies that the warm-up is not repeated on subsequent reconciles
@Karman580 Karman580 force-pushed the fix/rule-cache-cold-start-race branch from f9aba8b to 5c83d8f Compare May 14, 2026 14:05
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 14, 2026
@Karman580
Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: bootstrap-only taint enforcement silently skipped for nodes that join during the initial rule cache warm-up window

2 participants