feat(observability): split into composable per-component tasks

[scotwells]

Summary

• Split the monolithic observability stack into per-component kustomization directories and Taskfile tasks, so downstream repos can install only the components they need (e.g. Victoria Metrics + OTel Collector for resource-metrics e2e tests)
• Preserved backward compatibility: install-observability is now a thin composite that calls all sub-tasks, producing the same result as before
• Added webhook retry logic to the OTel Collector task to handle the v1beta1 CRD race condition that causes intermittent failures

New tasks

Task	Description	Dependencies
install-prometheus-crds	Prometheus Operator CRDs	-
install-victoria-metrics	Victoria Metrics (vmagent + vmsingle)	install-prometheus-crds
install-otel-collector	OpenTelemetry Operator + Collector	-
install-grafana	Grafana Operator + instance + datasources	install-victoria-metrics
install-loki	Loki for log aggregation	-
install-tempo	Tempo for distributed tracing	-
install-observability	Full stack (unchanged interface)	all of the above

Structure

Each component has its own kustomization.yaml under components/observability/<component>/ that is self-contained (includes namespace + helm-repositories). The root kustomization.yaml references individual files from subdirectories to avoid resource duplication.

components/observability/
├── kustomization.yaml              # root - references all components (backward compat)
├── namespace.yaml                  # shared
├── helm-repositories.yaml          # shared
├── prometheus-crds/
├── victoria-metrics/
├── otel-collector/
├── grafana/
│   └── datasources/
├── loki/
└── tempo/

Test plan

• kustomize build components/observability produces the same resource set as before
• kustomize build components/observability/<component> works for each component individually
• task --list shows all new tasks
• task install-victoria-metrics install-otel-collector deploys only VM + OTel on a constrained cluster
• task install-observability still deploys the full stack

🤖 Generated with Claude Code

[scotwells]

Follow-up commit b9515a4 addresses both problems from the review.

Problem 1 — removed duplicated shared files

• Deleted root-level namespace.yaml, helm-repositories.yaml, and datasources/ (they're now only in the components that own them).
• datasources/ lives under grafana/datasources/ (the Grafana instance owns the datasource CRs).
• Each component's helm-repositories.yaml now only contains the single repo it actually consumes. To let the root kustomize build compose everything without duplicate-resource errors, Loki and Tempo use distinct HelmRepository names (loki-charts, tempo-charts) — same URL, different Kubernetes object — and their HRs reference those names.
• Dropped prometheus-crds/namespace.yaml (CRDs are cluster-scoped; kustomization verified).
• Root components/observability/kustomization.yaml now references only the six component subdirectories.

Problem 2 — per-component namespaces

Each component deploys to its own namespace so kubectl delete ns <x> cleanly uninstalls it:

Component	Namespace
prometheus-crds	(cluster-scoped)
victoria-metrics	victoria-metrics-system
otel-collector	otel-collector-system
loki	loki-system
tempo	tempo-system
grafana	grafana-system

Cross-component references are now FQDNs:

• Grafana datasources → vmsingle-victoria-metrics-system-vm.victoria-metrics-system.svc.cluster.local:8428, vmalertmanager-…:9093, loki-system-loki.loki-system.svc.cluster.local:3100, tempo-system-tempo.tempo-system.svc.cluster.local:3100.
• OTel Collector exporters (otlphttp/tempo, prometheusremotewrite, otlphttp/loki) use the new FQDNs.
• VMAlert's datasource, notifier, remoteWrite URLs updated.
• defaultDashboards.grafanaOperator.allowCrossNamespaceImport flipped to true so VM-created dashboards can target the Grafana CR in grafana-system.

Taskfile's kubectl wait calls updated for the new namespaces (vmagent/vmsingle in victoria-metrics-system, otel-collector-collector DaemonSet in otel-collector-system). README refreshed.

Verification

• kustomize build components/observability → OK
• kustomize build components/observability/<each> → OK for all six

Diff stats

git diff main...HEAD --shortstat is now 31 files changed, 261 insertions(+), 89 deletions(-) — down from ~394 additions. Most of the remaining delta is the Taskfile expansion (necessary for the new composable tasks) and the new README.md section.

[scotwells]

Stacked fix in b3453bd: flipped defaultDashboards.grafanaOperator.enabled to false in the VM HelmRelease and added a kustomize patch in the root components/observability to flip it back to true for full-stack installs; standalone install-victoria-metrics no longer requires the grafana-operator CRDs.