ProactivePolicy and ClusterProactivePolicy do not evaluate workloads owned by cluster-scoped resources

[scubadam]

ProactivePolicy and ClusterProactivePolicy do not evaluate workloads owned by cluster-scoped resources

Version: automation-controller:0.1.3

Summary

When a Kubernetes workload's ownerReference chain terminates at a cluster-scoped resource (e.g. GatewayClass), the kubex automation engine silently skips it. Neither a namespace-scoped ProactivePolicy nor a ClusterProactivePolicy is able to rightsize these workloads.

Ownership chain

Pod
 └── ReplicaSet
      └── Deployment (in namespace: envoy-gateway-system)
           └── GatewayClass/envoy-gateway-class  ← cluster-scoped, no namespace

What we tried

Namespace-scoped ProactivePolicy

A ProactivePolicy in envoy-gateway-system targeting these Deployments was created and resolved (AutomationStrategyResolved: True). The engine never logged any evaluation or rightsizing activity for these pods, consistent with the expected behaviour that namespace-scoped policies cannot reach workloads whose ownerRef chain escapes the namespace.

Cluster-scoped ClusterProactivePolicy

A ClusterProactivePolicy was created with the following scope variations (tested iteratively):

# Attempt 1 — namespaceSelector matching the workload namespace
scope:
  namespaceSelector:
    operator: In
    values: [envoy-gateway-system]
  labelSelector:
    matchLabels:
      app.kubernetes.io/component: proxy
      app.kubernetes.io/managed-by: envoy-gateway

# Attempt 2 — permissive namespaceSelector (workaround for cluster-scoped owner)
scope:
  namespaceSelector:
    operator: NotIn
    values: [__no_match__]
  labelSelector:
    matchExpressions:
      - key: app.kubernetes.io/component
        operator: In
        values: [proxy]
      - key: app.kubernetes.io/managed-by
        operator: In
        values: [envoy-gateway]

# Attempt 3 — label added to GatewayClass, selector targeting it
# (GatewayClass labeled: app.kubernetes.io/managed-by=envoy-gateway)
scope:
  namespaceSelector:
    operator: NotIn
    values: [__no_match__]
  labelSelector:
    matchExpressions:
      - key: app.kubernetes.io/managed-by
        operator: In
        values: [envoy-gateway]

In all cases the policy resolved successfully (AutomationStrategyResolved: True) but the engine produced no admission webhook log entries and no workload annotation syncs for the proxy Deployments. Pod restarts were used to force admission webhook evaluation — confirmed via engine logs that the webhook fired for other pods in the same cluster at the same time.

Confirmed working baseline

A ClusterProactivePolicy with the same structure targeting the envoy-gateway controller Deployment (which has no escaping ownerRef — Helm-managed, no ownerReferences) worked correctly and produced RESIZED results immediately.

Expected behaviour

ClusterProactivePolicy should be able to evaluate and rightsize workloads whose ownerRef chain terminates at a cluster-scoped resource such as GatewayClass.

Questions

1. Is ClusterProactivePolicy expected to support cluster-scoped root owners? If so, is there a supported configuration?
2. Does the engine walk the ownerReferences chain and use the root owner's namespace/labels for scope resolution? If so, how should cluster-scoped owners be handled?
3. Is there a planned fix or workaround?

[julsemaan]

I'll attempt to replicate it. What should happen is that the Deployment is in scope and it should not care if the topmost owner is not in scope as long as an owner is in scope. So that sounds definitely like a bug.

[julsemaan]

I was not able to replicate this on v0.1.3

These are the manifests that I have for my envoy gateway

---
apiVersion: rightsizing.kubex.ai/v1alpha1
kind: AutomationStrategy
metadata:
  name: web-gateway-automation-strategy
  namespace: envoy-gateway-system
spec: {}
---
apiVersion: rightsizing.kubex.ai/v1alpha1
kind: ProactivePolicy
metadata:
  name: envoy-web-gateway-proactive-policy
  namespace: envoy-gateway-system
spec:
  automationStrategyRef:
    name: web-gateway-automation-strategy
  scope:
    labelSelector:
      matchLabels:
        gateway.envoyproxy.io/owning-gateway-name: web-gateway
    workloadTypes:
      - Deployment

And the annotations for the recommendations to apply appear as expected:

$ kubectl get deployment -o yaml  -n envoy-gateway-system \
| yq '.items[]
      | {
          name: .metadata.name,
          annotations: (
            .metadata.annotations
            | with_entries(select(.value | test("kubex.ai")))
          )
        }
      | select(.annotations != {})'
{
  "name": "envoy-default-web-gateway-6a499eb1",
  "annotations": {
    "proactive.rightsizing.kubex.ai/desired-resource-limits": "{\"policyName\":\"envoy-web-gateway-proactive-policy\",\"policyNamespace\":\"envoy-gateway-system\",\"policyAPIVersion\":\"rightsizing.kubex.ai/v1alpha1\",\"policyKind\":\"ProactivePolicy\",\"automationStrategyName\":\"web-gateway-automation-strategy\",\"automationStrategyNamespace\":\"envoy-gateway-system\",\"automationStrategyKind\":\"AutomationStrategy\",\"automationStrategyAPIVersion\":\"rightsizing.kubex.ai/v1alpha1\",\"containers\":{\"envoy\":{\"cpu\":\"400m\",\"memory\":\"1Gi\"},\"shutdown-manager\":{\"cpu\":\"100m\",\"memory\":\"128Mi\"}}}",
    "proactive.rightsizing.kubex.ai/desired-resource-requests": "{\"policyName\":\"envoy-web-gateway-proactive-policy\",\"policyNamespace\":\"envoy-gateway-system\",\"policyAPIVersion\":\"rightsizing.kubex.ai/v1alpha1\",\"policyKind\":\"ProactivePolicy\",\"automationStrategyName\":\"web-gateway-automation-strategy\",\"automationStrategyNamespace\":\"envoy-gateway-system\",\"automationStrategyKind\":\"AutomationStrategy\",\"automationStrategyAPIVersion\":\"rightsizing.kubex.ai/v1alpha1\",\"containers\":{\"envoy\":{\"cpu\":\"200m\",\"memory\":\"768Mi\"},\"shutdown-manager\":{\"cpu\":\"50m\",\"memory\":\"64Mi\"}}}"
  }
}

And I do have resizing events in the logs and the envoy gateway pods are sized as expected.

Can you see if these annotations are appearing on your envoy gateway Deployment objects?

[scubadam]

Sadly I cannot get this to work.
Even if I remove the scope completely and let it do everything in the namespace, it only touches the envoy-gateway controller pods/deployment in that namespace. All the proxy deployment are just ignored.

The only obvious difference to me was the owner ref of the deployment. For the controller - none - no ownerRef.

For the gateway proxy deployments their ownerRef is

  ownerReferences:
  - apiVersion: gateway.networking.k8s.io/v1
    kind: GatewayClass
    name: envoy-gateway-class
    uid: some-uuid

Which is a cluster scoped CR not in this namespace.

Otherwise I can't understand why the deployments/pod are teated differently?

Here is the Proactive Policy spec

spec:
  automationStrategyRef:
    name: envoy-gateway-balanced
  safetyChecks:
    maxAnalysisAgeDays: 5
  scope:
    labelSelector:
      matchLabels:
        app.kubernetes.io/managed-by: envoy-gateway
    workloadTypes:
    - Deployment
  weight: 0
status:
  conditions:
  - lastTransitionTime: "2026-04-21T07:33:05Z"
    message: referenced automation strategy envoy-gateway-system/envoy-gateway-balanced
      (rightsizing.kubex.ai/v1alpha1, Kind=AutomationStrategy) resolved
    observedGeneration: 2
    reason: Resolved
    status: "True"
    type: AutomationStrategyResolved

and the automation strategy spec

spec:
  enablement:
    cpu:
      limits:
        ceiling: "12"
        downsize: true
        setFromUnspecified: false
        upsize: true
      requests:
        ceiling: "8"
        downsize: true
        floor: 250m
        setFromUnspecified: true
        upsize: true
    memory:
      limits:
        ceiling: 24Gi
        downsize: true
        floor: 256Mi
        setFromUnspecified: true
        upsize: true
      requests:
        ceiling: 16Gi
        downsize: true
        floor: 128Mi
        setFromUnspecified: true
        upsize: true
  inPlaceResize:
    containerRestart: true
    enabled: true
  podEviction:
    enabled: true
    retryPodDisruptionBudget: true
  safetyChecks:
    enableHpaFilter: true
    enableLimitRangeFilter: true
    enablePauseUntilAnnotationCheck: true
    enablePodLimitRangeFilter: true
    enableResourceQuotaFilter: true
    enableVpaFilter: true
    minCpuChangePercent: 15
    minMemoryChangePercent: 15
    minReadyDuration: 30s
    nodeCpuHeadroom: 15%
    nodeMemoryHeadroom: 512Mi
    requireNodeAllocatable: true
    requireOwnerPodsReady: true
    resizeRetryInterval: 2m
    respectWorkloadMaxUnavailable: true

To note, the
gateway.envoyproxy.io/owning-gateway-name
gateway.envoyproxy.io/owning-gateway-namespace
are different for the multiple deployments in the envoy-gateway-system namespace so I didn't use those as my selectors. But I have tried many selectors and I know them to be working as the similarly labeled controller pods get rightsized.
There are no vpa/hpa/quota/limitrange blocking them. This is evidenced by the controller deployment and its pods which do get rightsized.

The actual proxy deployment pods are completely ignroed. There are no annotations.
The engine logs shows only the controller deployment workload no mention of the other n deployments that should have been matched.

manager 2026-04-21T07:33:05Z    INFO    rightsizing summary    {"controller": "policyevaluation", "reconcileID": "a313d3e0-9bed-4b3f-adfb-1d3c7421ed3b", "pod": {"name":"envoy-gateway-544bf6995d-2l956","namespace":"envoy-gateway-system"}, "type": "rightsizingSummary", "namespaceName": "envoy-gatewa
y-system", "podOwner": {"name":"envoy-gateway","namespace":"envoy-gateway-system"}, "policy": "ProactivePolicy/envoy-gateway-balanced", "rightsizingResult": "RESIZED", "resizePlan": {"envoy-gateway":{"limits":{"cpu":"290m","memory":"1534Mi"},"requests":{"cpu":"250m","memory":"510Mi"}}}, "failedChe
cks": [], "appliedFilters": [], "summary": "Applied rightsizing requests [envoy-gateway:[cpu=250m, memory=510Mi]] limits [envoy-gateway:[cpu=290m, memory=1534Mi]] via in-place resize"}
manager 2026-04-21T07:38:57Z    INFO    policy-resource    validation for policy upon update    {"kind": "ProactivePolicy", "name": "envoy-gateway-balanced"}
manager 2026-04-21T07:38:57Z    INFO    policy-resource    validation for policy upon update    {"kind": "ProactivePolicy", "name": "envoy-gateway-balanced"}
manager 2026-04-21T07:38:57Z    INFO    policy-resource    validation for policy upon update    {"kind": "ProactivePolicy", "name": "envoy-gateway-balanced"}
manager 2026-04-21T07:38:57Z    INFO    synced workload annotations to pods    {"controller": "workloadrecommendation", "reconcileID": "e382173c-2f86-4763-9f52-3f4a57d84f1b", "workload": {"name":"envoy-gateway","namespace":"envoy-gateway-system"}, "gvk": "apps/v1, Kind=Deployment", "desiredRecomme
ndationCount": 0, "desiredControlAnnotationCount": 0, "podsMatched": 2, "podsUpdated": 2}
manager 2026-04-21T07:52:55Z    INFO    policy-resource    validation for policy upon update    {"kind": "ProactivePolicy", "name": "envoy-gateway-balanced"}

[julsemaan]

Do you see the envoy gateway containers in the Kubex UI? One theory could be that they are not analyzed properly in Kubex and result in not resource recommendation.

Happy to plan a direct call with you to get to the bottom of this, I'll have folks internally get us in touch via email if you're good with this.

[scubadam]

In short yes they show up, with recommendations, in the UI. Happy to meet. I'll reach out as I have a meeting with our account reps tomorrow actually.