Specific Kustomizations hang with "New reconciliation triggered"

[aclerici38]

flux --version
flux version 2.8.3

Since enabling server-side-apply on my HelmReleases, some Kustomizations have been getting stuck with the status New reconciliation triggered by GitRepository/flux-system/flux-system. For me, it is specifically my External-Secrets Kustomization, which applies a HelmRelease, OCIRepository, and GrafanaDashboard resource.

I am only able to reproduce this with the CancelHealthCheckOnNewRevision feature-gate and SSA enabled on the HelmRelease. It only gets triggered with multiple commits before all the healthChecks in my cluster timeout, triggering the CancelHealthCheckOnNewRevision featureGate. Note the external-secrets Kustomization in question is already reconciled before the commit triggering the featureGate. From the logs, my elementary analysis is that the new reconciliation after the healthchecks are cancelled just never gets triggered here.
When manually reconciling via the cli when stuck flux reconcile ks external-secrets, the Kustomization reconciles immediately.

My repo is structured with a Kustomization per HelmRelease, and it happens only on my External-Secrets Kustomization. Others I have talked to with similar repos have observed it on External-Secrets, Cloudnative-PG, Dragonfly-operator, and Volsync. The only similarities I can come up with between those apps is templated CRDs in the chart and common applications for other Kustomziations to set in dependsOn.

My deployment in-question is here https://github.com/aclerici38/home-ops/tree/main/kubernetes/apps/cluster-infra/external-secrets/app
(The Kustomization comes from a kustomize component https://github.com/aclerici38/home-ops/tree/main/kubernetes/components/ks)
I am deploying flux through the operator and chart for the fluxInstance
https://github.com/aclerici38/home-ops/tree/main/kubernetes/apps/flux-system

Reproduction steps:

1. set --feature-gates=CancelHealthCheckOnNewRevision=true on the kustomize controller
2. Deploy a Kustomization that applies a HelmRelease to install External-Secrets with CRDs enabled
3. Set spec.upgrade.serverSideApply to enabled
4. Commit to any part of the repo to trigger reconciliations
5. Make another commit before all Kustomizations are reconciled, triggering the CancelHealthCheckOnNewRevision
6. Observe External-secrets Kustomization stuck with New reconciliation triggered by GitRepository/flux-system/flux-system in the status
7. Run flux reconcile kustomization external-secrets
8. Observe a successful reconciliation immediately

Kustomization when stuck:

k get ks -n cluster-infra external-secrets -o yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  annotations:
    reconcile.fluxcd.io/requestedAt: "2026-03-15T11:09:35.364675-07:00"
  creationTimestamp: "2025-12-13T20:19:09Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 6
  labels:
    components.dormammu/ks: app
    kustomize.dormammu/app-vars: "true"
    kustomize.toolkit.fluxcd.io/name: cluster-infra
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: external-secrets
  namespace: cluster-infra
  resourceVersion: "120857546"
  uid: d87ac9a8-333d-4f95-a127-a11e6b043ed9
spec:
  commonMetadata:
    labels:
      app.kubernetes.io/name: external-secrets
      app.kubernetes.io/namespace: cluster-infra
  components:
  - ../../../../components/hr
  dependsOn: []
  force: false
  interval: 30m
  path: ./kubernetes/apps/cluster-infra/external-secrets/app
  postBuild:
    substitute:
      APP_URL: external-secrets.REDACTED
    substituteFrom:
    - kind: ConfigMap
      name: external-secrets-app-vars-8mt975ht5k
      optional: false
    - kind: Secret
      name: cluster-settings
      optional: true
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  targetNamespace: cluster-infra
  timeout: 3m
  wait: true
status:
  conditions:
  - lastTransitionTime: "2026-03-24T13:22:22Z"
    message: Running health checks for revision refs/heads/main@sha1:124a229e1226e2a91d04c13a75eb4520cb8f0839
      with a timeout of 3m0s
    observedGeneration: 6
    reason: ProgressingWithRetry
    status: "True"
    type: Reconciling
  - lastTransitionTime: "2026-03-24T13:22:22Z"
    message: New reconciliation triggered by GitRepository/flux-system/flux-system
    observedGeneration: 6
    reason: HealthCheckCanceled
    status: "False"
    type: Ready
  - lastTransitionTime: "2026-03-24T13:22:22Z"
    message: Running health checks for revision refs/heads/main@sha1:124a229e1226e2a91d04c13a75eb4520cb8f0839
      with a timeout of 3m0s
    observedGeneration: 6
    reason: Progressing
    status: Unknown
    type: Healthy
  history:
  - digest: sha256:f8a3ea6188bbb3496502e086c607474de8e345aa57a3df11f423342d1cfa5c88
    firstReconciled: "2026-03-20T19:29:32Z"
    lastReconciled: "2026-03-24T13:21:36Z"
    lastReconciledDuration: 278.288099ms
    lastReconciledStatus: ReconciliationSucceeded
    metadata:
      revision: refs/heads/main@sha1:f2a25a8beb1f6bf92091e4c2d87e60911553e5fd
    totalReconciliations: 212
  - digest: sha256:fcc55296250f7cea8d36486acc933275ee66c344faaf8ab20322f639ac3a6ec4
    firstReconciled: "2026-03-19T22:24:54Z"
    lastReconciled: "2026-03-20T18:59:45Z"
    lastReconciledDuration: 134.966415ms
    lastReconciledStatus: ReconciliationSucceeded
    metadata:
      revision: refs/heads/main@sha1:f5737db9fdaae67f81093e35fbc2f23f7e52b5e9
    totalReconciliations: 60
  - digest: sha256:1106cff91949f4aeb7e636e13f25e5aab3009998a66a275e4b6e28b3b941c234
    firstReconciled: "2026-03-16T18:51:06Z"
    lastReconciled: "2026-03-19T22:19:33Z"
    lastReconciledDuration: 114.31534ms
    lastReconciledStatus: ReconciliationSucceeded
    metadata:
      revision: refs/heads/main@sha1:02994e83fb13e08815cfbd954ede85589944b9fd
    totalReconciliations: 180
  - digest: sha256:f06297926ef50591f513d62340fc1980a215180815ec78e7e13a94f4d35bcddd
    firstReconciled: "2026-03-19T08:35:23Z"
    lastReconciled: "2026-03-19T21:09:53Z"
    lastReconciledDuration: 338.91764ms
    lastReconciledStatus: ReconciliationSucceeded
    metadata:
      revision: refs/heads/main@sha1:42140223327314998a0b1b44867be7a047db2aca
    totalReconciliations: 65
  - digest: sha256:2eb7d794a295d11fd693ec7344236922618dd2c05283e8dc577a9a1fa1ac427a
    firstReconciled: "2026-03-15T16:21:29Z"
    lastReconciled: "2026-03-16T18:35:16Z"
    lastReconciledDuration: 130.317971ms
    lastReconciledStatus: ReconciliationSucceeded
    metadata:
      revision: refs/heads/main@sha1:ef17f4878933bc497197cccf856983ccb4d88ac5
    totalReconciliations: 14
  inventory:
    entries:
    - id: cluster-infra_external-secrets_grafana.integreatly.org_GrafanaDashboard
      v: v1beta1
    - id: cluster-infra_external-secrets_helm.toolkit.fluxcd.io_HelmRelease
      v: v2
    - id: cluster-infra_external-secrets_source.toolkit.fluxcd.io_OCIRepository
      v: v1
  lastAppliedRevision: refs/heads/main@sha1:f2a25a8beb1f6bf92091e4c2d87e60911553e5fd
  lastAttemptedRevision: refs/heads/main@sha1:124a229e1226e2a91d04c13a75eb4520cb8f0839
  lastHandledReconcileAt: "2026-03-15T11:09:35.364675-07:00"
  observedGeneration: 6

HelmRelease when stuck:

k get hr -n cluster-infra external-secrets -o yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  annotations:
    reconcile.fluxcd.io/forceAt: "1772044083"
    reconcile.fluxcd.io/requestedAt: "1772044083"
  creationTimestamp: "2025-12-13T20:25:15Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 21
  labels:
    app.kubernetes.io/name: external-secrets
    app.kubernetes.io/namespace: cluster-infra
    kustomize.toolkit.fluxcd.io/name: external-secrets
    kustomize.toolkit.fluxcd.io/namespace: cluster-infra
  name: external-secrets
  namespace: cluster-infra
  resourceVersion: "120819928"
  uid: c9d3d662-00c5-4612-af13-3115c0a92c78
spec:
  chartRef:
    kind: OCIRepository
    name: external-secrets
  driftDetection:
    mode: enabled
  install:
    crds: CreateReplace
    disableTakeOwnership: true
    remediation:
      retries: 1
    serverSideApply: true
    strategy:
      name: RetryOnFailure
    timeout: 2m
  interval: 1h
  maxHistory: 1
  postRenderers:
  - kustomize:
      patches:
      - patch: |
          - op: add
            path: /spec/rules/0/timeouts
            value:
              request: 0s
        target:
          group: gateway.networking.k8s.io
          kind: HTTPRoute
  upgrade:
    cleanupOnFail: true
    crds: CreateReplace
    disableTakeOwnership: true
    remediation:
      retries: 1
    serverSideApply: enabled
    strategy:
      name: RemediateOnFailure
    timeout: 2m
  values:
    certController:
      image:
        repository: ghcr.io/external-secrets/external-secrets
      serviceMonitor:
        enabled: true
        interval: 1m
    image:
      repository: ghcr.io/external-secrets/external-secrets
    installCRDs: true
    leaderElect: true
    replicaCount: 2
    serviceMonitor:
      enabled: true
      interval: 1m
    webhook:
      image:
        repository: ghcr.io/external-secrets/external-secrets
      serviceMonitor:
        enabled: true
        interval: 1m
status:
  conditions:
  - lastTransitionTime: "2026-03-20T19:29:30Z"
    message: Helm upgrade succeeded for release cluster-infra/external-secrets.v16
      with chart external-secrets@2.2.0
    observedGeneration: 21
    reason: UpgradeSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-03-20T19:29:30Z"
    message: Helm upgrade succeeded for release cluster-infra/external-secrets.v16
      with chart external-secrets@2.2.0
    observedGeneration: 21
    reason: UpgradeSucceeded
    status: "True"
    type: Released
  history:
  - action: upgrade
    apiVersion: v2
    appVersion: v2.2.0
    chartName: external-secrets
    chartVersion: 2.2.0
    configDigest: sha256:cfbd9b63d5473e42e267b8433c69c43f8e88de14d7f752ef0752e3ae85feee69
    digest: sha256:ea2b41585e7816c88c44b034b3fa022f2e1f9e71fad544e4b06da76eb9ecd0f9
    firstDeployed: "2025-12-13T20:14:45Z"
    lastDeployed: "2026-03-20T19:28:57Z"
    name: external-secrets
    namespace: cluster-infra
    ociDigest: sha256:feb252e996e2ce31ea015a1e098f2c5d438389a0a2fc1f43659b3f5421046f6f
    status: deployed
    version: 16
  - apiVersion: v2
    appVersion: v2.1.0
    chartName: external-secrets
    chartVersion: 2.1.0
    configDigest: sha256:cfbd9b63d5473e42e267b8433c69c43f8e88de14d7f752ef0752e3ae85feee69
    digest: sha256:286ab5dd70b7bdbdb4572f957112782433691f43785011b25a81d5dfd1a6aeb3
    firstDeployed: "2025-12-13T20:14:45Z"
    lastDeployed: "2026-03-15T19:31:38Z"
    name: external-secrets
    namespace: cluster-infra
    ociDigest: sha256:447e6cdcc323b70bcc4f487672f7ca98ba581ba9b68789be87ce159a6caeed9c
    status: superseded
    version: 15
  inventory:
    entries:
    - id: cluster-infra_external-secrets-cert-controller__ServiceAccount
      v: v1
    - id: cluster-infra_external-secrets__ServiceAccount
      v: v1
    - id: cluster-infra_external-secrets-webhook__ServiceAccount
      v: v1
    - id: cluster-infra_external-secrets-webhook__Secret
      v: v1
    - id: _acraccesstokens.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _cloudsmithaccesstokens.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _clusterexternalsecrets.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _clustergenerators.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _clusterpushsecrets.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _clustersecretstores.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _ecrauthorizationtokens.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _externalsecrets.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _fakes.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _gcraccesstokens.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _generatorstates.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _githubaccesstokens.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _grafanas.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _mfas.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _passwords.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _pushsecrets.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _quayaccesstokens.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _secretstores.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _sshkeys.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _stssessiontokens.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _uuids.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _vaultdynamicsecrets.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _webhooks.generators.external-secrets.io_apiextensions.k8s.io_CustomResourceDefinition
      v: v1
    - id: _external-secrets-cert-controller_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _external-secrets-controller_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _external-secrets-view_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _external-secrets-edit_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _external-secrets-servicebindings_rbac.authorization.k8s.io_ClusterRole
      v: v1
    - id: _external-secrets-cert-controller_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: _external-secrets-controller_rbac.authorization.k8s.io_ClusterRoleBinding
      v: v1
    - id: cluster-infra_external-secrets-leaderelection_rbac.authorization.k8s.io_Role
      v: v1
    - id: cluster-infra_external-secrets-leaderelection_rbac.authorization.k8s.io_RoleBinding
      v: v1
    - id: cluster-infra_external-secrets-cert-controller-metrics__Service
      v: v1
    - id: cluster-infra_external-secrets-metrics__Service
      v: v1
    - id: cluster-infra_external-secrets-webhook__Service
      v: v1
    - id: cluster-infra_external-secrets-cert-controller_apps_Deployment
      v: v1
    - id: cluster-infra_external-secrets_apps_Deployment
      v: v1
    - id: cluster-infra_external-secrets-webhook_apps_Deployment
      v: v1
    - id: _secretstore-validate_admissionregistration.k8s.io_ValidatingWebhookConfiguration
      v: v1
    - id: _externalsecret-validate_admissionregistration.k8s.io_ValidatingWebhookConfiguration
      v: v1
    - id: cluster-infra_external-secrets-metrics_monitoring.coreos.com_ServiceMonitor
      v: v1
    - id: cluster-infra_external-secrets-webhook-metrics_monitoring.coreos.com_ServiceMonitor
      v: v1
    - id: cluster-infra_external-secrets-cert-controller-metrics_monitoring.coreos.com_ServiceMonitor
      v: v1
  lastAttemptedConfigDigest: sha256:cfbd9b63d5473e42e267b8433c69c43f8e88de14d7f752ef0752e3ae85feee69
  lastAttemptedGeneration: 21
  lastAttemptedReleaseAction: upgrade
  lastAttemptedReleaseActionDuration: 33.83738943s
  lastAttemptedRevision: 2.2.0
  lastAttemptedRevisionDigest: sha256:feb252e996e2ce31ea015a1e098f2c5d438389a0a2fc1f43659b3f5421046f6f
  lastHandledForceAt: "1772044083"
  lastHandledReconcileAt: "1772044083"
  observedGeneration: 21
  observedPostRenderersDigest: sha256:50499434250e928b35e1730acd29adba98d922b80f733141437b97eb1b2eaa53
  storageNamespace: cluster-infra

kustomize-controller full logs:
ks.txt

helm-controller full logs:
hr.txt

I took a small screen recording in case my description wasn't clear, showing manual reconciliation works imediately
https://github.com/user-attachments/assets/4acda8b5-4f97-4d37-a7be-7518a34e4c1f

Thanks!

[chrede88]

I have also seen this happen on my cluster with SSA. It happened to a Rook-Ceph kustomization in my case. I didn't gather any logs, so can't really contribute with anything other than this.

[ajgon]

This may not be limited to SSA only - I also stumble into this issue from time to time, and I'm not using SSA in my HelmReleases.

[mircea-pavel-anton]

It regularly happens in my cluster as well. The kustomizations also seem to always be the same: volsync, cloudnative PG, external secrets

I'll get to my computer in the next couple of days and see if I can get some logs or something more useful

[matheuscscp]

I'd like to start by ruling out SSA on helm-controller having a direct impact on kustomize-controller's watches:

• Kustomizations (.spec changes or flux reconcile ks)
• Referenced OCIRepositories (.status.artifact.revision changes)
• Referenced GitRepositories (.status.artifact.revision changes)
• Referenced Buckets (.status.artifact.revision changes)
• Referenced ExternalArtifacts (.status.artifact.revision changes)
• Referenced ConfipMaps (.metadata.resourceVersion changes)
• Referenced Secrets (.metadata.resourceVersion changes)

Health check cancellation happens for the Kustomizations actively being reconciled that are concurrently enqueued due to one of the watch events above. The only way for SSA on helm-controller to have an impact here is: a HelmRelease applies one of the objects above and triggers the health check cancellation of a related Kustomization. But the error message is New reconciliation triggered by GitRepository/flux-system/flux-system, which means that the Git repo artifact revision changed while the Kustomization was being reconciled. This tells me that SSA had nothing to do with the cancellation.

Is everyone here running Flux 2.8.3? Has everyone here enabled the CancelHealthCheckOnNewRevision feature gate on kustomize-controller and helm-controller? Are your Flux Kustomizations actually getting stuck after a cancellation?

@aclerici38 @chrede88 @ajgon @mircea-pavel-anton

[chrede88]

@matheuscscp yes, I'm running flux 2.8.3 now, although I'm not 100% sure I was back when I last saw this issue. But for sure flux 2.8.x!
I do have the CancelHealthCheckOnNewRevision feature gate set on the kustomize controller, but for some reason not on the helm controller.
Yes, the kustomization displaying New reconciliation triggered by GitRepository/flux-system/flux-system won't do anything until I manually kick it and start a new reconcile with flux reconcile ks.

[matheuscscp]

@chrede88 How many times did you see this in total? Do you have deterministic steps to reproduce?

[chrede88]

@matheuscscp I've only seen this a couple of times and not recently. I think both @mircea-pavel-anton & @aclerici38 have a better grasp on this.

[mircea-pavel-anton]

I'm back at my computer and I have done a deeper investigation of this in my cluster (with assistance from Claude) and wanted to share the findings in case they help narrow down the root cause.

Environment

• Flux version: 2.8.x (kustomize-controller image: ghcr.io/fluxcd/kustomize-controller:v1.8.0)
• Feature gate: --feature-gates=CancelHealthCheckOnNewRevision=true (explicitly enabled https://github.com/mirceanton/home-ops/blob/main/apps/flux-system/flux-instance/app/helm-release.yaml#L134-L141)
• --concurrent=10, --requeue-dependency=5s (in case this matters)

Full deployment configuration can be found here: https://github.com/mirceanton/home-ops/blob/main/apps/flux-system/flux-instance/app/helm-release.yaml

SSA Not Required

I'd like to start by ruling out SSA on helm-controller having a direct impact on kustomize-controller's watches

The two KSes that got stuck in my cluster this time are database-system/cloudnative-pg-operator and storage-system/volsync. I checked both of their HelmReleases and neither has spec.upgrade.serverSideApply: true.

This confirms what @ajgon noted earlier that SSA is not a prerequisite. The feature gate + a new revision arriving during health checks seems to be sufficient.

Steps to reproduce

How many times did you see this in total?

A lot. When I push commits in "quick" succession, this always happens, reliably.

Do you have deterministic steps to reproduce?

The race condition seems to require:

1. CancelHealthCheckOnNewRevision=true is enabled.
2. A Kustomization that manages an operator (CRDs + workloads) enters health-check phase. These tend to run longer because the controller waits for CRD-backed resources to become healthy.
3. A second commit is pushed to the GitRepository before the health check completes.
4. The feature gate fires, cancels the in-flight health check, and sets Ready: False / reason: HealthCheckCanceled.
5. The controller does not re-enqueue the KS for reconciliation against the new revision.

The fact that volsync and cloudnative-pg-operator are particularly susceptible makes sense since they deploy operators with CRDs, so health checks take longer than a typical app, giving a wider window for the race.

Observed cluster state

Both stuck KSes show identical conditions:

conditions:
  - type: Ready
    status: "False"
    reason: HealthCheckCanceled
    message: "New reconciliation triggered by GitRepository/flux-system/flux-system"
    lastTransitionTime: "2026-03-28T15:12:17Z"

  - type: Reconciling
    status: "True"
    reason: ProgressingWithRetry
    message: "Running health checks for revision refs/heads/main@sha1:77fae531... with a timeout of 9m30s"

  - type: Healthy
    status: "Unknown"
    reason: Progressing

lastAttemptedRevision: refs/heads/main@sha1:77fae531321e606feaf6c48e1a7faeccb3c2acae  # new commit — never applied
lastAppliedRevision:   refs/heads/main@sha1:a45871d85679abca743b5bd6714d4102e3e321fc  # old commit — stuck here

Kubernetes events at the time of cancellation:

database-system   Normal  HealthCheckCanceled  kustomization/cloudnative-pg-operator
  Health checks canceled due to new reconciliation triggered by GitRepository/flux-system/flux-system

storage-system    Normal  HealthCheckCanceled  kustomization/volsync
  Health checks canceled due to new reconciliation triggered by GitRepository/flux-system/flux-system

[stefanprodan]

@mircea-pavel-anton do these Kustomization that get stuck have dependsOn? It may be that the reconciliation following the cancelation is rejected due to dependencies not being ready.

[mircea-pavel-anton]

@stefanprodan they do not:

• volsync: https://github.com/mirceanton/home-ops/blob/main/apps/storage-system/volsync/app.ks.yaml#L21
• cnpg: https://github.com/mirceanton/home-ops/blob/main/apps/database-system/cloudnative-pg-operator/app.ks.yaml#L21

[matheuscscp]

• Flux version: 2.8.x (kustomize-controller image: ghcr.io/fluxcd/kustomize-controller:v1.8.0)

Then #1614 is eliminated (it's only in v1.8.2)

[aclerici38]

I must have been wrong about SSA then, oops

Is everyone here running Flux 2.8.3?

flux version
flux: v2.8.3
distribution: flux-v2.8.3
helm-controller: v1.5.3@sha256:b150af0cd7a501dafe2374b1d22c39abf0572465df4fa1fb99b37927b0d95d75
kustomize-controller: v1.8.2@sha256:c480b89e26e42f6c112a4f683244a7979de3a2ca299bed7d5367ddf4fed706f0
notification-controller: v1.8.2@sha256:87806dc20caff40b37280ea3155cc9ef3e995402997c49a8f9f9c6bff57e1499
source-controller: v1.8.1@sha256:7382d002cffeed2d877331353f95797e89c0aa7ecb432e661eeeda3e590b3293

Has everyone here enabled the CancelHealthCheckOnNewRevision feature gate on kustomize-controller and helm-controller? Are your Flux Kustomizations actually getting stuck after a cancellation?

To be completely honest I didn't know the feature gate existed for the helm-controller, but it was set on the kustomize-controller and the Kustomizations would get stuck until a manual reconcile; I never waited long enough to see if they truly didn't get requeued, I just know I have found them stuck and a manual reconcile fixes it immediately.

Looks like I missed the feature gate with flux v2.8, I will monitor with it also set on the helm-controller. Though, when the Kustomizations get stuck any child HelmReleases are in the reconciled state as shown in my original post.

[matheuscscp]

@stefanprodan We assumed we would not need to requeue because the cancellation is already triggered by a requeue, but that is apparently not very reliable. To solve this, I think we could just use RequeueAfter with the value of --requeue-dependency. For the cases when our assumption is true (probably the vast majority), KC and HC would now reconcile twice after the cancellation. A RequeueAfter would not cause a second cancellation, because it is not a watch event (only watch events cause cancellation).

[stefanprodan]

To solve this, I think we could just use RequeueAfter

A RequeueAfter should have no effect if the object is already in the queue, controller-runtime will skip the request. So I'm not sure how would this solve anything.

KC and HC would now reconcile twice after the cancellation

This would be really bad, almost no one uses ArtifactGenerator, which means we'll reconcile the whole desired state twice, 2nd time doing a dry-run of potentially thousands of objects.