Skip to content

fix: install PaymentMethod CRDs + restore webhook Service name#41

Merged
mattdjenkinson merged 1 commit into
mainfrom
fix/payment-method-crd-webhook-service-regen
May 22, 2026
Merged

fix: install PaymentMethod CRDs + restore webhook Service name#41
mattdjenkinson merged 1 commit into
mainfrom
fix/payment-method-crd-webhook-service-regen

Conversation

@mattdjenkinson
Copy link
Copy Markdown
Contributor

Summary

Two release-process bugs that landed in v0.2.0 and still exist on main — together they take down billing-controller-manager and break BillingAccount admission against the deployed bundle in staging.

Bug 1 — PaymentMethod / PaymentMethodClass CRDs in the bundle but never applied

`config/base/crd/kustomization.yaml` only lists four CRDs even though controller-gen had emitted six. `PaymentMethod` and `PaymentMethodClass` exist in `config/base/crd/bases/` but were never referenced, so the bundle that ships to Flux installs only the four originals.

Live impact in staging:

```
ERROR controller-runtime.source.Kind if kind is a CRD, it should be installed before calling Start
{"kind": "PaymentMethod.billing.miloapis.com", "error": "no matches for kind "PaymentMethod" in version "billing.miloapis.com/v1alpha1""}
ERROR Could not wait for Cache to sync ... timed out waiting for cache to be synced for Kind *v1alpha1.PaymentMethod
```

`billing-controller-manager` exits 1, CrashLoopBackOff, 13 restarts on each replica.

Bug 2 — Webhook Service name regression

`config/base/webhook/manifests.yaml` ships with the kubebuilder placeholder Service name (`webhook-service`) instead of the `billing-webhook` Service this repo actually declares. Kustomize's `nameReference` transformer cannot bridge the two because nothing ever renames the Service — they are simply different identifiers.

The deployed `MutatingWebhookConfiguration` therefore points at `webhook-service.billing-system.svc`, which does not exist, and every `BillingAccount` mutating-admission call fails DNS lookup:

```
BillingAccount/milo-system/fake-usage dry-run failed (InternalError):
Internal error occurred: failed calling webhook "mbillingaccount.kb.io":
failed to call webhook: Post "https://webhook-service.billing-system.svc:443/mutate-billing-miloapis-com-v1alpha1-billingaccount?timeout=10s":
dial tcp: lookup webhook-service.billing-system.svc on 10.2.0.10:53: no such host
```

The existing remediation was a comment in `webhook/kustomization.yaml` asking devs to hand-restore the service name after each `task manifests` regeneration. v0.2.0 demonstrates how reliable that is.

Fix

File Change
`config/base/crd/kustomization.yaml` Add the two missing CRD entries.
`config/base/webhook/manifests.yaml` Restore the eight `name: webhook-service` references to `name: billing-webhook`.
`Taskfile.yaml` (`manifests` task) Append an in-place `perl -pi` step so the next regeneration is idempotent. The footgun comment in `webhook/kustomization.yaml` is replaced with one that documents the new automated fix-up.

Validation

  • `kustomize build config/base/crd` lists all six CRDs.
  • `kustomize build config/base/webhook` clean.
  • Re-ran `task manifests` end-to-end: `grep -c "name: billing-webhook"` → 8, `grep -c "name: webhook-service"` → 0 in the regenerated file.

Deployment recovery

Once this lands and a new bundle tag publishes, FluxCD will reconcile `billing-milo-control-plane` (CRDs land), `billing-milo-webhook` (MWC Service ref corrected), and `billing-manager` (Deployment becomes healthy once its CRD watches succeed).

Two release-process bugs that landed in v0.2.0 and stuck around on
main; together they take down the billing-controller-manager and
break BillingAccount admission against the deployed bundle:

1. config/base/crd/kustomization.yaml only listed four CRDs even
   though controller-gen had emitted six. PaymentMethod and
   PaymentMethodClass were in config/base/crd/bases/ but never
   referenced, so the bundle that ships to Flux installed only the
   four originals. The controller-manager then tried to watch
   PaymentMethod, hit `no matches for kind "PaymentMethod" in
   version "billing.miloapis.com/v1alpha1"`, timed out waiting for
   the cache to sync, and exited 1. Result: CrashLoopBackOff on the
   manager replicas in staging.

2. config/base/webhook/manifests.yaml shipped with the kubebuilder
   placeholder Service name (`webhook-service`) instead of the
   billing-webhook Service this repo actually declares. Kustomize's
   nameReference transformer cannot bridge the two because nothing
   ever renames a Service — they are simply different identifiers.
   The deployed MutatingWebhookConfiguration therefore pointed at
   webhook-service.billing-system.svc, which does not exist, and
   every BillingAccount mutating-admission call failed DNS lookup.

For (2) the existing fix-up was a comment in webhook/kustomization.yaml
asking devs to hand-restore the service name after each `task manifests`
regeneration. v0.2.0 demonstrates how reliable that is. Wire the
fix-up into the manifests task itself so the regenerated file is
always consistent with the declared Service.
@mattdjenkinson mattdjenkinson merged commit a32c675 into main May 22, 2026
7 checks passed
@mattdjenkinson mattdjenkinson deleted the fix/payment-method-crd-webhook-service-regen branch May 22, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants