Skip to content

Add support for drift detection ignore rules#1627

Draft
dipti-pai wants to merge 1 commit intofluxcd:mainfrom
dipti-pai:drift-ignore-rules
Draft

Add support for drift detection ignore rules#1627
dipti-pai wants to merge 1 commit intofluxcd:mainfrom
dipti-pai:drift-ignore-rules

Conversation

@dipti-pai
Copy link
Copy Markdown
Member

Changes include :

  • Spec changes to add DriftIgnoreRules[] to kustomization spec.
  • Convert the drift ignore rules to []jsondiff.IgnoreRule and set in ApplyOptions
  • Doc changes and tests.

Test summary with kustomize-controller

E2E Test Results — DriftIgnoreRules

Setup

$ kubectl get gitrepository -n drift-ignore-test
NAME              URL                                           AGE   READY   STATUS
drift-test-repo   https://github.com/dipti-pai/flux-test-repo   17s   True    stored artifact for revision 'main@sha1:b2a96870abdf6a295501e298a147e1e4ea5d878a'

Source manifests (in ssaDriftIgnoreRulesTest/) deploy:

  • ConfigMap test-configmanaged-key: "v1", ignored-key: "original", another-ignored: "keep-me"
  • Deployment test-nginxnginx:1.25, 1 replica, with resource requests/limits
  • Service test-service — with annotation external-dns.alpha.kubernetes.io/hostname: "test.example.com"

Reconciliation verification approach

Each test triggers reconciliation after introducing drift, checks the controller logs and confirms non-ignored fields are in-tact.


Test 1 — Ignore /spec/replicas on Deployments (HPA use case)

Kustomization:

driftIgnoreRules:
  - paths:
      - "/spec/replicas"
    target:
      kind: Deployment

Initial state:

$ kubectl get ks -n drift-ignore-test
NAME                AGE   READY   STATUS
test-basic-ignore   29s   True    Applied revision: main@sha1:b2a96870abdf6a295501e298a147e1e4ea5d878a

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.replicas}'
1

Simulate HPA scaling (server-side apply with hpa-controller field manager):

$ kubectl apply --server-side --field-manager=hpa-controller --force-conflicts -f deployment-scaled.yaml
deployment.apps/test-nginx serverside-applied

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.replicas}'
3

Trigger reconciliation:

$ kubectl annotate ks test-basic-ignore -n drift-ignore-test reconcile.fluxcd.io/requestedAt=$(date +%s) --overwrite
kustomization.kustomize.toolkit.fluxcd.io/test-basic-ignore annotated

After reconciliation — resource verification:

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.replicas}'
3  # <-- replicas preserved!

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].image}'
nginx:1.25  # <-- image unchanged

** controller log:**

{
  "level": "info",
  "ts": "2026-03-31T17:44:58.182Z",
  "msg": "server-side apply completed",
  "Kustomization": {"name": "test-basic-ignore", "namespace": "drift-ignore-test"},
  "reconcileID": "745d39ee-dca1-4654-b608-449d29b54a48",
  "output": {
    "ConfigMap/drift-ignore-test/test-config": "unchanged",
    "Deployment/drift-ignore-test/test-nginx": "configured",
    "Namespace/drift-ignore-test": "unchanged",
    "Service/drift-ignore-test/test-service": "unchanged"
  }
}

Test 2 — Targeted selectors (external-dns + ConfigMap field)

Kustomization:

driftIgnoreRules:
  - paths:
      - "/metadata/annotations/external-dns.alpha.kubernetes.io~1hostname"
    target:
      kind: Service
      name: test-service
  - paths:
      - "/data/ignored-key"
    target:
      kind: ConfigMap
      name: test-config

Initial state:

$ kubectl get svc test-service -n drift-ignore-test -o jsonpath='{.metadata.annotations}'
{"external-dns.alpha.kubernetes.io/hostname":"test.example.com"}

$ kubectl get cm test-config -n drift-ignore-test -o jsonpath='{.data}'
{"another-ignored":"keep-me","ignored-key":"original","managed-key":"v1"}

Simulate external-dns and controller changes (server-side apply):

$ kubectl apply --server-side --field-manager=external-dns --force-conflicts -f svc-modified.yaml
service/test-service serverside-applied

$ kubectl apply --server-side --field-manager=config-controller --force-conflicts -f cm-modified.yaml
configmap/test-config serverside-applied

$ kubectl get svc test-service -n drift-ignore-test -o jsonpath='{.metadata.annotations}'
{"external-dns.alpha.kubernetes.io/hostname":"production.example.com"}

$ kubectl get cm test-config -n drift-ignore-test -o jsonpath='{.data}'
{"another-ignored":"keep-me","ignored-key":"changed-by-controller","managed-key":"v1"}

After reconciliation — resource verification:

$ kubectl get svc test-service -n drift-ignore-test -o jsonpath='{.metadata.annotations}'
{"external-dns.alpha.kubernetes.io/hostname":"production.example.com"}  # <-- annotation preserved!

$ kubectl get cm test-config -n drift-ignore-test -o jsonpath='{.data}'
{"another-ignored":"keep-me","ignored-key":"changed-by-controller","managed-key":"v1"}  # <-- ignored-key preserved!

controller log:

{
  "level": "info",
  "ts": "2026-03-31T17:47:36.973Z",
  "msg": "server-side apply completed",
  "Kustomization": {"name": "test-targeted-ignore", "namespace": "drift-ignore-test"},
  "reconcileID": "072c53f3-47b8-465d-8788-01fe2266e6aa",
  "output": {
    "ConfigMap/drift-ignore-test/test-config": "configured",
    "Deployment/drift-ignore-test/test-nginx": "unchanged",
    "Namespace/drift-ignore-test": "unchanged",
    "Service/drift-ignore-test/test-service": "configured"
  }
}

Test 3 — Multi-resource (Deployment replicas + Service annotations + ConfigMap fields)

Kustomization:

driftIgnoreRules:
  - paths:
      - "/spec/replicas"
    target:
      kind: Deployment
  - paths:
      - "/metadata/annotations"
    target:
      kind: Service
  - paths:
      - "/data/ignored-key"
      - "/data/another-ignored"
    target:
      kind: ConfigMap

Out-of-band changes via server-side apply (3 different field managers):

$ kubectl apply --server-side --field-manager=hpa-controller --force-conflicts ...     # replicas: 3
$ kubectl apply --server-side --field-manager=external-dns --force-conflicts ...       # added annotation
$ kubectl apply --server-side --field-manager=config-controller --force-conflicts ...  # changed CM keys

Before reconciliation:

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.replicas}'
3

$ kubectl get svc test-service -n drift-ignore-test -o jsonpath='{.metadata.annotations}'
{"external-dns.alpha.kubernetes.io/hostname":"test.example.com","my-custom-annotation":"my-value"}

$ kubectl get cm test-config -n drift-ignore-test -o jsonpath='{.data}'
{"another-ignored":"y","ignored-key":"x","managed-key":"v1"}

After reconciliation — resource verification:

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.replicas}'
3  # <-- preserved

$ kubectl get svc test-service -n drift-ignore-test -o jsonpath='{.metadata.annotations}'
{"external-dns.alpha.kubernetes.io/hostname":"test.example.com","my-custom-annotation":"my-value"}  # <-- preserved

$ kubectl get cm test-config -n drift-ignore-test -o jsonpath='{.data}'
{"another-ignored":"y","ignored-key":"x","managed-key":"v1"}  # <-- ignored fields preserved, managed-key intact

** controller log:**

{
  "level": "info",
  "ts": "2026-03-31T17:50:30.122Z",
  "msg": "server-side apply completed",
  "Kustomization": {"name": "test-multi-resource", "namespace": "drift-ignore-test"},
  "reconcileID": "76976031-dc4a-4763-a6d1-d0b4adaa9d44",
  "output": {
    "ConfigMap/drift-ignore-test/test-config": "configured",
    "Deployment/drift-ignore-test/test-nginx": "configured",
    "Namespace/drift-ignore-test": "unchanged",
    "Service/drift-ignore-test/test-service": "unchanged"
  }
}

Test 4 — Nested paths (VPA container resources)

Kustomization:

driftIgnoreRules:
  - paths:
      - "/spec/template/spec/containers/0/resources"
    target:
      kind: Deployment
      name: test-nginx

Initial state:

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].resources}'
{"limits":{"cpu":"200m","memory":"256Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}

Simulate VPA adjusting resources (server-side apply with vpa-recommender field manager):

$ kubectl apply --server-side --field-manager=vpa-recommender --force-conflicts -f deployment-vpa.yaml
deployment.apps/test-nginx serverside-applied

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].resources}'
{"limits":{"cpu":"1","memory":"512Mi"},"requests":{"cpu":"250m","memory":"256Mi"}}

After reconciliation — VPA resources preserved:

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].resources}'
{"limits":{"cpu":"1","memory":"512Mi"},"requests":{"cpu":"250m","memory":"256Mi"}}  # <-- VPA resources preserved!

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].image}'
nginx:1.25  # <-- image unchanged

controller log (VPA change):

{
  "level": "info",
  "ts": "2026-03-31T17:53:07.760Z",
  "msg": "server-side apply completed",
  "Kustomization": {"name": "test-nested-paths", "namespace": "drift-ignore-test"},
  "reconcileID": "61099129-e120-4b06-8643-0e3fe4f2df51",
  "output": {
    "ConfigMap/drift-ignore-test/test-config": "unchanged",
    "Deployment/drift-ignore-test/test-nginx": "configured",
    "Namespace/drift-ignore-test": "unchanged",
    "Service/drift-ignore-test/test-service": "unchanged"
  }
}

Test 4b — Non-ignored field drift correction

Change image to nginx:1.24, then reconcile:

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].image}'
nginx:1.24  # <-- changed

# After reconciliation:
$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].image}'
nginx:1.25  # <-- reverted to nginx:1.25

$ kubectl get deploy test-nginx -n drift-ignore-test -o jsonpath='{.spec.template.spec.containers[0].resources}'
{"limits":{"cpu":"1","memory":"512Mi"},"requests":{"cpu":"250m","memory":"256Mi"}}  # <-- VPA resources STILL preserved

controller log (image correction):

{
  "level": "info",
  "ts": "2026-03-31T17:54:34.124Z",
  "msg": "server-side apply completed",
  "Kustomization": {"name": "test-nested-paths", "namespace": "drift-ignore-test"},
  "reconcileID": "271aa062-20eb-4146-aaf6-da883e243f19",
  "output": {
    "ConfigMap/drift-ignore-test/test-config": "unchanged",
    "Deployment/drift-ignore-test/test-nginx": "configured",
    "Namespace/drift-ignore-test": "unchanged",
    "Service/drift-ignore-test/test-service": "unchanged"
  }
}

Test 5 — Edge cases (non-existent paths + non-matching selector)

Kustomization:

driftIgnoreRules:
  - paths:
      - "/data/nonexistent-key"
      - "/spec/this/does/not/exist"
  - paths:
      - "/data/ignored-key"
    target:
      kind: ConfigMap
      name: does-not-exist

Initial state — reconciles successfully despite non-existent paths:

$ kubectl get ks -n drift-ignore-test
NAME              AGE   READY   STATUS
test-edge-cases   29s   True    Applied revision: main@sha1:b2a96870abdf6a295501e298a147e1e4ea5d878a

Change ignored-key via SSA (rule targets does-not-exist CM, so should NOT match test-config):

# Before reconciliation:
$ kubectl get cm test-config -n drift-ignore-test -o jsonpath='{.data.ignored-key}'
edge-changed

# After reconciliation:
$ kubectl get cm test-config -n drift-ignore-test -o jsonpath='{.data.ignored-key}'
original  # <-- reverted to "original" (selector targets "does-not-exist", not "test-config")

controller log:

{
  "level": "info",
  "ts": "2026-03-31T17:57:09.475Z",
  "msg": "server-side apply completed",
  "Kustomization": {"name": "test-edge-cases", "namespace": "drift-ignore-test"},
  "reconcileID": "2daa7063-36f1-46af-9c43-6cb674e399c8",
  "output": {
    "ConfigMap/drift-ignore-test/test-config": "configured",
    "Deployment/drift-ignore-test/test-nginx": "unchanged",
    "Namespace/drift-ignore-test": "unchanged",
    "Service/drift-ignore-test/test-service": "unchanged"
  }
}

No errors in controller logs:

$ kubectl -n kustomize-system logs deployment/kustomize-controller --since=2m | grep -i error
(no errors)

Test 6 — Ignore mandatory /spec/selector, introduce drift, no other SSA owner

Kustomization:

driftIgnoreRules:
  - paths:
      - "/spec/selector"
    target:
      kind: Deployment
      name: test-nginx

Initial creation succeeds:

$ kubectl get ks test-6c-selector-no-owner -n drift-test-10
NAME                        AGE   READY   STATUS
test-6c-selector-no-owner   28s   True    Applied revision: main@sha1:b2a96870abdf6a295501e298a147e1e4ea5d878a

$ kubectl get deploy test-nginx -n drift-test-10 -o jsonpath='{.spec.selector}'
{"matchLabels":{"app":"nginx"}}

Introduce image drift via non-SSA patch (no new manager claims selector):

$ kubectl patch deploy test-nginx -n drift-test-10 --type json \
    -p '[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"nginx:1.24"}]'
deployment.apps/test-nginx patched

$ kubectl get deploy test-nginx -n drift-test-10 -o jsonpath='{.spec.template.spec.containers[0].image}'
nginx:1.24  # <-- introduce drift

After reconciliation — APPLY FAILS:

$ kubectl get ks test-6c-selector-no-owner -n drift-test-10
NAME                        AGE    READY   STATUS
test-6c-selector-no-owner   100s   False   Deployment/drift-test-10/test-nginx apply failed: ...

Controller error log:

{
  "level": "error",
  "ts": "2026-03-31T22:06:05.184Z",
  "msg": "Reconciliation failed after 206.673654ms, next try in 2m0s",
  "Kustomization": {"name": "test-6c-selector-no-owner", "namespace": "drift-test-10"},
  "error": "Deployment/drift-test-10/test-nginx apply failed: Deployment.apps \"test-nginx\" is invalid: [spec.selector: Required value, spec.template.metadata.labels: Invalid value: {\"app\":\"nginx\"}: `selector` does not match template `labels`, spec.selector: Invalid value: null: field is immutable]"
}

Signed-off-by: Dipti Pai <diptipai89@outlook.com>
@dipti-pai dipti-pai force-pushed the drift-ignore-rules branch from 029fa79 to 02e1e73 Compare April 1, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant