Skip to content

CP-37371: Use distroless Prometheus image by default#648

Open
evan-cz wants to merge 1 commit intodevelopfrom
CP-37371
Open

CP-37371: Use distroless Prometheus image by default#648
evan-cz wants to merge 1 commit intodevelopfrom
CP-37371

Conversation

@evan-cz
Copy link
Contributor

@evan-cz evan-cz commented Feb 3, 2026

Prometheus 3.10.0 introduced an official distroless image variant that eliminates shells, package managers, and other utilities from the container. This significantly reduces the attack surface of the Prometheus container, addressing security concerns around the nodes/proxy RBAC permission (see docs/wiki/The-nodes-proxy-Problem.md).

Functional Change:

Before: The Helm chart defaulted to the standard Prometheus image (busybox-based, includes shell and utilities) at version v3.7.3. The Prometheus image tag was resolved by the cloudzero-agent.prometheusImageTag helper with an inline fallback chain duplicated across templates.

After: The Helm chart defaults to the distroless Prometheus image at version v3.10.0. The image tag is resolved by a new cloudzero-agent.Values.components.prometheus.image.tag helper that appends "-distroless" to Chart.AppVersion when no explicit tag is set. The deprecated server.image.tag compat override is handled by generateImage's compat layer, not the helper itself.

Solution:

  1. Bumped Chart.AppVersion from v3.7.3 to v3.10.0 in helm/Chart.yaml

  2. Added cloudzero-agent.Values.components.prometheus.image.tag helper in helm/templates/_helpers.tpl that resolves components.prometheus.image.tag with a fallback to {Chart.AppVersion}-distroless

  3. Removed the old cloudzero-agent.prometheusImageTag helper and updated all call sites (agent-deploy.yaml, agent-daemonset.yaml, prometheusAgentFlag) to use the new helper

  4. Removed outdated inline comments about version-specific distroless logic from agent-deploy.yaml and agent-daemonset.yaml

  5. Regenerated Helm template test snapshots (alloy.yaml, cert-manager.yaml, federated.yaml, istio.yaml, manifest.yaml) to reflect the new default image tag

Validation:

  • Helm template snapshots regenerated and verified

@evan-cz evan-cz requested a review from a team as a code owner February 3, 2026 13:30
@evan-cz evan-cz changed the base branch from develop to dependabot/docker/docker/golang-1.25.6-alpine February 3, 2026 13:30
Base automatically changed from dependabot/docker/docker/golang-1.25.6-alpine to develop February 3, 2026 18:09
@evan-cz evan-cz force-pushed the CP-37371 branch 5 times, most recently from 1e0648c to 1ab04a5 Compare March 4, 2026 15:53
@evan-cz evan-cz changed the title CP-37371: bundle Prometheus binary in CloudZero Agent image CP-37371: Use distroless Prometheus image by default Mar 4, 2026
@evan-cz evan-cz enabled auto-merge March 4, 2026 16:03
# Disabled: marocchino/validate-dependabot@v3 fails due to a schemastore.org
# schema change (base.json#/definitions/timezone reference can't be resolved).
# See https://json.schemastore.org/dependabot-2.0.json
if: github.event_name == 'disabled'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really part of this, but the action is failing and this uncorks CI

Prometheus 3.10.0 introduced an official distroless image variant that
eliminates shells, package managers, and other utilities from the container.
This significantly reduces the attack surface of the Prometheus container,
addressing security concerns around the nodes/proxy RBAC permission
(see docs/wiki/The-nodes-proxy-Problem.md).

Functional Change:

Before: The Helm chart defaulted to the standard Prometheus image
(busybox-based, includes shell and utilities) at version v3.7.3. The
Prometheus image tag was resolved by the `cloudzero-agent.prometheusImageTag`
helper with an inline fallback chain duplicated across templates.

After: The Helm chart defaults to the distroless Prometheus image at
version v3.10.0. The image tag is resolved by a new
`cloudzero-agent.Values.components.prometheus.image.tag` helper that
appends "-distroless" to Chart.AppVersion when no explicit tag is set.
The deprecated server.image.tag compat override is handled by
generateImage's compat layer, not the helper itself.

Solution:

1. Bumped Chart.AppVersion from v3.7.3 to v3.10.0 in helm/Chart.yaml

2. Added `cloudzero-agent.Values.components.prometheus.image.tag` helper
   in helm/templates/_helpers.tpl that resolves
   components.prometheus.image.tag with a fallback to
   `{Chart.AppVersion}-distroless`

3. Removed the old `cloudzero-agent.prometheusImageTag` helper and updated
   all call sites (agent-deploy.yaml, agent-daemonset.yaml,
   prometheusAgentFlag) to use the new helper

4. Removed outdated inline comments about version-specific distroless
   logic from agent-deploy.yaml and agent-daemonset.yaml

5. Regenerated Helm template test snapshots (alloy.yaml, cert-manager.yaml,
   federated.yaml, istio.yaml, manifest.yaml) to reflect the new default
   image tag

Validation:

- Helm template snapshots regenerated and verified
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant