fix: install PaymentMethod CRDs + restore webhook Service name#41
Merged
mattdjenkinson merged 1 commit intoMay 22, 2026
Merged
Conversation
Two release-process bugs that landed in v0.2.0 and stuck around on main; together they take down the billing-controller-manager and break BillingAccount admission against the deployed bundle: 1. config/base/crd/kustomization.yaml only listed four CRDs even though controller-gen had emitted six. PaymentMethod and PaymentMethodClass were in config/base/crd/bases/ but never referenced, so the bundle that ships to Flux installed only the four originals. The controller-manager then tried to watch PaymentMethod, hit `no matches for kind "PaymentMethod" in version "billing.miloapis.com/v1alpha1"`, timed out waiting for the cache to sync, and exited 1. Result: CrashLoopBackOff on the manager replicas in staging. 2. config/base/webhook/manifests.yaml shipped with the kubebuilder placeholder Service name (`webhook-service`) instead of the billing-webhook Service this repo actually declares. Kustomize's nameReference transformer cannot bridge the two because nothing ever renames a Service — they are simply different identifiers. The deployed MutatingWebhookConfiguration therefore pointed at webhook-service.billing-system.svc, which does not exist, and every BillingAccount mutating-admission call failed DNS lookup. For (2) the existing fix-up was a comment in webhook/kustomization.yaml asking devs to hand-restore the service name after each `task manifests` regeneration. v0.2.0 demonstrates how reliable that is. Wire the fix-up into the manifests task itself so the regenerated file is always consistent with the declared Service.
JoseSzycho
approved these changes
May 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two release-process bugs that landed in v0.2.0 and still exist on
main— together they take downbilling-controller-managerand break BillingAccount admission against the deployed bundle in staging.Bug 1 — PaymentMethod / PaymentMethodClass CRDs in the bundle but never applied
`config/base/crd/kustomization.yaml` only lists four CRDs even though controller-gen had emitted six. `PaymentMethod` and `PaymentMethodClass` exist in `config/base/crd/bases/` but were never referenced, so the bundle that ships to Flux installs only the four originals.
Live impact in staging:
```
ERROR controller-runtime.source.Kind if kind is a CRD, it should be installed before calling Start
{"kind": "PaymentMethod.billing.miloapis.com", "error": "no matches for kind "PaymentMethod" in version "billing.miloapis.com/v1alpha1""}
ERROR Could not wait for Cache to sync ... timed out waiting for cache to be synced for Kind *v1alpha1.PaymentMethod
```
`billing-controller-manager` exits 1, CrashLoopBackOff, 13 restarts on each replica.
Bug 2 — Webhook Service name regression
`config/base/webhook/manifests.yaml` ships with the kubebuilder placeholder Service name (`webhook-service`) instead of the `billing-webhook` Service this repo actually declares. Kustomize's `nameReference` transformer cannot bridge the two because nothing ever renames the Service — they are simply different identifiers.
The deployed `MutatingWebhookConfiguration` therefore points at `webhook-service.billing-system.svc`, which does not exist, and every `BillingAccount` mutating-admission call fails DNS lookup:
```
BillingAccount/milo-system/fake-usage dry-run failed (InternalError):
Internal error occurred: failed calling webhook "mbillingaccount.kb.io":
failed to call webhook: Post "https://webhook-service.billing-system.svc:443/mutate-billing-miloapis-com-v1alpha1-billingaccount?timeout=10s":
dial tcp: lookup webhook-service.billing-system.svc on 10.2.0.10:53: no such host
```
The existing remediation was a comment in `webhook/kustomization.yaml` asking devs to hand-restore the service name after each `task manifests` regeneration. v0.2.0 demonstrates how reliable that is.
Fix
Validation
Deployment recovery
Once this lands and a new bundle tag publishes, FluxCD will reconcile `billing-milo-control-plane` (CRDs land), `billing-milo-webhook` (MWC Service ref corrected), and `billing-manager` (Deployment becomes healthy once its CRD watches succeed).