⏲ Add initContainers to jobs to wait for dependencies instead of crashing (demos)

[NickLarsenNZ]

Note

This came out of the 25.3.0 Release Retro

Many demos have (Job or other) pods showing CrashLoopBackoff until their dependencies are satisfied.

This causes confusion when launching demos, especially during demo testing.

Each stack/demo should make use of initContainers to wait for dependencies to be satisfied without giving false failure indications.

Tip

• Ensure a ServiceAccount is created and attached to applicable Pods, and give it permissions to watch Jobs/Pods so it can wait.
• Ensure the ServiceAccount doesn't specify a namespace so that the demo can be installed into any namespace.
• Use namespace scoped RBAC (Role/RoleBinding instead of ClusterRole/ClusterRoleBinding).

List of demos

List generation command

cat demos/demos-v2.yaml | yq -yr '.demos | keys' | sed 's/^-/- [ ]/'

• airflow-scheduled-job
• data-lakehouse-iceberg-trino-spark
• end-to-end-security
• hbase-hdfs-load-cycling-data
• jupyterhub-keycloak
• jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data
• logging
• nifi-kafka-druid-earthquake-data
• nifi-kafka-druid-water-level-data
• signal-processing
• spark-k8s-anomaly-detection-taxi-data
• trino-iceberg
• trino-taxi-data

Warning

It is possible that some stacks also need the same treatment, but most of the waiting can be done in the actual demo.