Summary
Alert rules are currently maintained as handwritten YAML files while recording rules are generated from the Jsonnet mixin. Moving alerts into the mixin would make them consistent and enable reusable alert templates.
Current state
| File |
Source |
Recording rules (activity-recordings.yaml) |
Generated from Jsonnet mixin |
All alert rules (activity-alerts.yaml, vector-alerts.yaml, dlq-alerts.yaml, slo-alerts.yaml, clickhouse-alerts.yaml) |
Handwritten YAML |
AlertmanagerConfig (alertmanager-config.yaml) |
Handwritten YAML |
Benefits of moving to the mixin
- Consistent build pipeline — all observability config generated from one source
- Reusable alert templates via the existing
observability/lib/alerts.libsonnet library
- Easier to enforce naming conventions and label consistency
- Single
task observability:build-mixin command regenerates everything
- Reduces drift between alert definitions and recording rules they reference
Tradeoff
Alert changes would require running the Jsonnet build pipeline instead of directly editing YAML.
Summary
Alert rules are currently maintained as handwritten YAML files while recording rules are generated from the Jsonnet mixin. Moving alerts into the mixin would make them consistent and enable reusable alert templates.
Current state
activity-recordings.yaml)activity-alerts.yaml,vector-alerts.yaml,dlq-alerts.yaml,slo-alerts.yaml,clickhouse-alerts.yaml)alertmanager-config.yaml)Benefits of moving to the mixin
observability/lib/alerts.libsonnetlibrarytask observability:build-mixincommand regenerates everythingTradeoff
Alert changes would require running the Jsonnet build pipeline instead of directly editing YAML.