enableHpaFilter does not filter workloads managed by a memory-based HPA

[scubadam]

Bug: enableHpaFilter does not filter workloads managed by a memory-based HPA

Component: kubex-automation-engine / AutomationStrategy safety checks (spec.safetyChecks.enableHpaFilter)

Severity: High — automation engine actively resizes HPA-managed workloads it should skip

---

Description

enableHpaFilter: true fails to filter a workload that has an HPA with a memory utilization target. The filter is set, the HPA exists and is actively managing the workload, yet kubex generates and applies resize recommendations regardless.

---

Environment

• kubex-automation-engine: 0.1.1
• Kubernetes: EKS (v1.31)

---

Steps to Reproduce

1. Create a StatefulSet (my-hpa-app) in a namespace (my-namespace)
2. Create an HPA targeting it with targetMemoryUtilizationPercentage set — the Kubernetes API server normalises scaleTargetRef.kind to lowercase for StatefulSet targets regardless of what is submitted
3. Create an AutomationStrategy with safetyChecks.enableHpaFilter: true
4. Create a ProactivePolicy scoping the StatefulSet

---

Observed Behaviour

The automation engine ignores the HPA and generates resize recommendations for the workload. Pod annotations confirm active rightsizing is being applied:

"proactive.rightsizing.kubex.ai/desired-resource-limits": {
  "my-hpa-app": { "cpu": "42870m", "memory": "43430Mi" }
}

Confirming the filter is enabled in the live strategy:

$ kubectl get automationstrategy my-strategy -n my-namespace \
    -o jsonpath='{.spec.safetyChecks.enableHpaFilter}'
true

Confirming the HPA exists, targets the workload, and has a memory utilization target:

$ kubectl get hpa my-hpa-app -n my-namespace -o jsonpath='{.spec.scaleTargetRef}'
{"apiVersion":"apps/v1","kind":"statefulset","name":"my-hpa-app"}

$ kubectl get hpa my-hpa-app -n my-namespace -o jsonpath='{.spec.metrics}'
[{"resource":{"name":"memory","target":{"averageUtilization":60,"type":"Utilization"}},"type":"Resource"}]

---

Expected Behaviour

enableHpaFilter: true should prevent kubex from resizing workloads whose memory is already managed by an HPA — modifying memory requests directly changes the HPA trigger threshold and undermines horizontal scaling behaviour.

---

Possible Cause

Inspecting the HPA, the scaleTargetRef.kind is lowercase (statefulset):

scaleTargetRef:
  apiVersion: apps/v1
  kind: statefulset     # lowercase
  name: my-hpa-app

We attempted to patch this to StatefulSet (PascalCase) to test the theory, but the Kubernetes API server normalises the value back to lowercase on write — it cannot be changed. This is not a helm chart quirk; it is how the Kubernetes HPA API stores scaleTargetRef.kind for StatefulSet targets.

It is likely the HPA filter performs a case-sensitive match on scaleTargetRef.kind and fails to recognise statefulset as matching the StatefulSet workload. Since this normalisation is enforced by the API server, the filter must handle lowercase kinds — there is no client-side workaround.

We cannot confirm this is the root cause without access to the engine source, but it is the most likely explanation given the evidence.

---

VPA Filter Comparison

enableVpaFilter appears unaffected — VPA objects in the same namespace use PascalCase targetRef.kind and are correctly filtered. This is consistent with a case-sensitivity issue specific to HPA matching.

---

Workaround

Explicitly exclude HPA-managed workloads from the ProactivePolicy scope using a NotIn label selector rather than relying on enableHpaFilter:

scope:
  labelSelector:
    matchExpressions:
      - key: app.kubernetes.io/name
        operator: NotIn
        values:
          - my-hpa-app

[julsemaan]

Thanks @scubadam for reporting the issue. We were able to replicate and address it. It will make its way through in v0.2.0. Will tag this ticket in the release PR.

Also fwiw, we were unable to replicate the lowercase enforcement from the apiserver and case was respected in our manifests.

[scubadam]

Sure, np re case, it was just a possible cause as VPA filtering worked and claude noticed that possible difference between reference naming between hpa and vpa.

Can I see also the release of:
#67
will be in that 0.2.0 release ? I am eagerly awaiting this as I have some guaranteed QoS pods which need scaling depending on cluster but should really maintain guaranteed QoS and I had just noticed this feature in the docs and tried it to realise the CRD chart had not been updated.
Thanks for the prompt responses. So far its been quite smooth overall with the new automation-engine. Look forward to being able to share the namespaced CRDs with tenants on clusters to enable them to leverage this themselves.

[julsemaan]

Glad to hear that it's been going well overall.

#67 is planned for v0.2.0, we need to do some internal qualification of the release and it should go out. Hopefully mid next-week or before