feat: mtls enable

README.md

@@ -11,7 +11,8 @@ $  helm install  kube-prometheus-stack kube-prometheus-stack
 
 To deploy Gthulhu using Helm charts, follow these steps:
 ```bash
-$ helm install gthulhu gthulhu -f ./gthulhu/values-production.yaml
+$ cd gthulhu
+$ helm install gthulhu . -f ./values-production.yaml --set mtls.enabled=true --set-file mtls.ca.cert=certs/ca.crt --set-file mtls.dm.cert=certs/dm.crt --set-file mtls.dm.key=certs/dm.key --set-file mtls.manager.cert=certs/manager.crt --set-file mtls.manager.key=certs/manager.key
 ```
 
 To uninstall Gthulhu, run the following command:

gthulhu/.helmignore

@@ -21,3 +21,7 @@
 .idea/
 *.tmproj
 .vscode/
+
+# Generated mTLS certificates
+certs/
+gen-mtls-certs.sh

gthulhu/README.md

@@ -214,6 +214,150 @@ kubectl logs -l app.kubernetes.io/component=api
 helm uninstall gthulhu
 ```
 
+## mTLS (Mutual TLS)
+
+Gthulhu supports **mutual TLS** to authenticate and encrypt traffic on two communication paths:
+
+| Path | Client | Server | Notes |
+|------|--------|--------|-------|
+| Manager → DM sidecar | Manager (Deployment) | DM sidecar (DaemonSet) | Cross-node; protects scheduling intents |
+| Scheduler → DM sidecar | Scheduler (same Pod) | DM sidecar (same Pod) | Loopback; protects the local API call |
+
+Both paths share a single **private CA** — every certificate is signed by this CA so each peer can verify the other.
+
+### Quick Start
+
+```bash
+# 1. Generate a private CA + leaf certificates
+./gen-mtls-certs.sh certs
+
+# 2. Install the chart with mTLS enabled
+helm install gthulhu ./gthulhu \
+  --set mtls.enabled=true \
+  --set-file mtls.ca.cert=certs/ca.crt \
+  --set-file mtls.dm.cert=certs/dm.crt \
+  --set-file mtls.dm.key=certs/dm.key \
+  --set-file mtls.manager.cert=certs/manager.crt \
+  --set-file mtls.manager.key=certs/manager.key
+```
+
+### Using Your Own Certificates
+
+If you already have a PKI or want to bring your own certificates, follow the steps below.
+
+#### 1. Create a Private CA
+
+```bash
+# EC P-256 key (recommended); RSA-4096 also works
+openssl ecparam -name prime256v1 -genkey -noout -out ca.key
+
+# Self-signed CA certificate (10-year validity)
+openssl req -new -x509 -days 3650 \
+  -key ca.key -out ca.crt \
+  -subj "/CN=Gthulhu-Private-CA"
+```
+
+#### 2. Generate the DM Sidecar Server Certificate
+
+The DM sidecar is the TLS **server**. Its certificate needs a `subjectAltName` that covers
+`localhost` and `127.0.0.1` (so the in-pod scheduler can connect) plus any DNS names
+the Manager uses to reach it (e.g. `*.svc.cluster.local`).
+
+```bash
+openssl ecparam -name prime256v1 -genkey -noout -out dm.key
+
+openssl req -new -key dm.key -out dm.csr \
+  -subj "/CN=gthulhu-decisionmaker"
+
+openssl x509 -req -days 730 \
+  -in dm.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
+  -extfile <(printf "subjectAltName=DNS:localhost,IP:127.0.0.1,DNS:*.svc.cluster.local\nextendedKeyUsage=serverAuth,clientAuth") \
+  -out dm.crt
+```
+
+#### 3. Generate the Manager / Scheduler Client Certificate
+
+The Manager and the Scheduler both act as mTLS **clients** when talking to the DM sidecar.
+They share the same client certificate.
+
+```bash
+openssl ecparam -name prime256v1 -genkey -noout -out manager.key
+
+openssl req -new -key manager.key -out manager.csr \
+  -subj "/CN=gthulhu-manager"
+
+openssl x509 -req -days 730 \
+  -in manager.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
+  -extfile <(printf "extendedKeyUsage=clientAuth") \
+  -out manager.crt
+```
+
+#### 4. Supply the Certificates to the Chart
+
+**Option A — inline via `--set-file`:**
+
+```bash
+helm install gthulhu ./gthulhu \
+  --set mtls.enabled=true \
+  --set-file mtls.ca.cert=ca.crt \
+  --set-file mtls.dm.cert=dm.crt \
+  --set-file mtls.dm.key=dm.key \
+  --set-file mtls.manager.cert=manager.crt \
+  --set-file mtls.manager.key=manager.key
+```
+
+**Option B — pre-created Kubernetes Secret:**
+
+```bash
+kubectl create secret generic my-gthulhu-mtls \
+  --from-file=ca.crt \
+  --from-file=dm.crt \
+  --from-file=dm.key \
+  --from-file=manager.crt \
+  --from-file=manager.key
+
+helm install gthulhu ./gthulhu \
+  --set mtls.enabled=true \
+  --set mtls.existingSecret=my-gthulhu-mtls
+```
+
+### Certificate Rotation
+
+Because the ConfigMap and the scheduler mTLS config Secret are created with
+`immutable: true`, you must use `helm upgrade --force` (which deletes and
+recreates immutable resources) when rotating certificates:
+
+```bash
+helm upgrade gthulhu ./gthulhu --force \
+  --set mtls.enabled=true \
+  --set-file mtls.ca.cert=new-ca.crt \
+  --set-file mtls.dm.cert=new-dm.crt \
+  --set-file mtls.dm.key=new-dm.key \
+  --set-file mtls.manager.cert=new-manager.crt \
+  --set-file mtls.manager.key=new-manager.key
+```
+
+### mTLS Configuration Reference
+
+| Parameter | Description | Default |
+|-----------|-------------|---------|
+| `mtls.enabled` | Enable mutual TLS | `false` |
+| `mtls.existingSecret` | Name of a pre-created Secret containing all PEM files | `""` |
+| `mtls.ca.cert` | PEM-encoded CA certificate | `""` |
+| `mtls.dm.cert` | PEM-encoded DM sidecar server certificate | `""` |
+| `mtls.dm.key` | PEM-encoded DM sidecar server private key | `""` |
+| `mtls.manager.cert` | PEM-encoded Manager/Scheduler client certificate | `""` |
+| `mtls.manager.key` | PEM-encoded Manager/Scheduler client private key | `""` |
+
+### Architecture Notes
+
+- The Manager's **external HTTP API** (web GUI / Ingress) remains **plain HTTP**.
+  Use a Kubernetes Ingress with TLS termination for external HTTPS.
+- When mTLS is enabled, health-check probes on the DM sidecar switch from
+  `httpGet` to `tcpSocket` because the kubelet cannot present a client certificate.
+- The scheduler config is stored in a Kubernetes **Secret** (not a ConfigMap)
+  when mTLS is enabled, because it contains private-key material inline.
+
 ## Development
 
 ### Testing the Chart

gthulhu/certs/ca.crt

@@ -0,0 +1,11 @@
+-----BEGIN CERTIFICATE-----
+MIIBjzCCATWgAwIBAgIUSJ2qiDFEvwKXlPAGWOnaD3onTGQwCgYIKoZIzj0EAwIw
+HTEbMBkGA1UEAwwSR3RodWxodS1Qcml2YXRlLUNBMB4XDTI2MDIyMzAzMTM0NFoX
+DTM2MDIyMTAzMTM0NFowHTEbMBkGA1UEAwwSR3RodWxodS1Qcml2YXRlLUNBMFkw
+EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsrU+DZyBpiuB5pvSuS/fbOGckLlwTC73
+mkQ7hnWwDnA/q+baN77PVtoVZrGNwxxgD0FAhzdrkRAf2jfEuTiqfqNTMFEwHQYD
+VR0OBBYEFIMViDVVNYHUrpQd8MIRbGMk+zjUMB8GA1UdIwQYMBaAFIMViDVVNYHU
+rpQd8MIRbGMk+zjUMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSAAwRQIg
+Ro9LhiUYzT7F5iXEJ0IM64quGDGlSWrbiRBO2odpTq0CIQDMEBxBhsqlx7h4ok6D
+vfoIMENsvJMMzAwN8uA6Y61lKA==
+-----END CERTIFICATE-----

gthulhu/certs/ca.key

@@ -0,0 +1,5 @@
+-----BEGIN EC PRIVATE KEY-----
+MHcCAQEEIIzkO4hC30h7tkNNeCt/NUej6NYWubQ1N6D7vef8uX9yoAoGCCqGSM49
+AwEHoUQDQgAEsrU+DZyBpiuB5pvSuS/fbOGckLlwTC73mkQ7hnWwDnA/q+baN77P
+VtoVZrGNwxxgD0FAhzdrkRAf2jfEuTiqfg==
+-----END EC PRIVATE KEY-----

gthulhu/certs/dm.crt

@@ -0,0 +1,16 @@
+-----BEGIN CERTIFICATE-----
+MIICdjCCAhygAwIBAgIUCT9p4TDp3lc2Fwp+EfecoNsTVVgwCgYIKoZIzj0EAwIw
+HTEbMBkGA1UEAwwSR3RodWxodS1Qcml2YXRlLUNBMB4XDTI2MDIyMzAzMTM0NVoX
+DTI4MDIyMzAzMTM0NVowIDEeMBwGA1UEAwwVZ3RodWxodS1kZWNpc2lvbm1ha2Vy
+MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEI9rucwma2o8H2eLPg+DLOlMqxEYp
+rOWqFw6xdWrNt6FYcRpbvPeO1R75ef86pmQzdIt8N87kvRpk1UMsXW8McqOCATUw
+ggExMIHPBgNVHREEgccwgcSCCWxvY2FsaG9zdIcEfwAAAYI9Ki5ndGh1bGh1LWd0
+aHVsaHUtc2NoZWR1bGVyLXNpZGVjYXIuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2Nh
+bIIvKi5ndGh1bGh1LWd0aHVsaHUtc2NoZWR1bGVyLXNpZGVjYXIuZGVmYXVsdC5z
+dmOCO2d0aHVsaHUtZ3RodWxodS1zY2hlZHVsZXItc2lkZWNhci5kZWZhdWx0LnN2
+Yy5jbHVzdGVyLmxvY2FshwQKAQAbMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEF
+BQcDAjAdBgNVHQ4EFgQU+5CDmEm5uoIDQ+cMpwVIaLhVk1kwHwYDVR0jBBgwFoAU
+gxWINVU1gdSulB3wwhFsYyT7ONQwCgYIKoZIzj0EAwIDSAAwRQIhALAeHhtuVmV9
+runXu4ssKoxuH5EbeeGaQPgSFn+rDzTbAiByGECW4rDlc7EtHtj3YoFBjDPIJ5RM
+YLrD4+pcyXXXpA==
+-----END CERTIFICATE-----

gthulhu/certs/dm.key

@@ -0,0 +1,5 @@
+-----BEGIN EC PRIVATE KEY-----
+MHcCAQEEIMEPRqDJaZ1/86TCIOWL9TxxIbh/y+LJPi7IUj7nZ1BooAoGCCqGSM49
+AwEHoUQDQgAEI9rucwma2o8H2eLPg+DLOlMqxEYprOWqFw6xdWrNt6FYcRpbvPeO
+1R75ef86pmQzdIt8N87kvRpk1UMsXW8Mcg==
+-----END EC PRIVATE KEY-----

gthulhu/certs/manager.crt

@@ -0,0 +1,11 @@
+-----BEGIN CERTIFICATE-----
+MIIBkDCCATagAwIBAgIUCT9p4TDp3lc2Fwp+EfecoNsTVVkwCgYIKoZIzj0EAwIw
+HTEbMBkGA1UEAwwSR3RodWxodS1Qcml2YXRlLUNBMB4XDTI2MDIyMzAzMTM0NVoX
+DTI4MDIyMzAzMTM0NVowGjEYMBYGA1UEAwwPZ3RodWxodS1tYW5hZ2VyMFkwEwYH
+KoZIzj0CAQYIKoZIzj0DAQcDQgAEc6EOY5fjmiO/ffE3yHcQBnDINRAE1GiBk0xc
+eRny/6dELBcWO48TbSXw4uu6YD3zczk2gq5jKKSFM8WUUmSxVaNXMFUwEwYDVR0l
+BAwwCgYIKwYBBQUHAwIwHQYDVR0OBBYEFHtY+SaE/DxkUtwY2AlWdk/PeoD4MB8G
+A1UdIwQYMBaAFIMViDVVNYHUrpQd8MIRbGMk+zjUMAoGCCqGSM49BAMCA0gAMEUC
+IHQugTGHFp986lKCEleY0aRgw5xU46eRgLOVhw20GzxDAiEAlk8BmsmPXQTrsIA9
+n9cGKdtJjHC91Lb/RTydrSH73w4=
+-----END CERTIFICATE-----

gthulhu/certs/manager.key

@@ -0,0 +1,5 @@
+-----BEGIN EC PRIVATE KEY-----
+MHcCAQEEIMb/LITD2EAFHDfryINaeuh/cVgLDac36J4weDztgkOyoAoGCCqGSM49
+AwEHoUQDQgAEc6EOY5fjmiO/ffE3yHcQBnDINRAE1GiBk0xceRny/6dELBcWO48T
+bSXw4uu6YD3zczk2gq5jKKSFM8WUUmSxVQ==
+-----END EC PRIVATE KEY-----

gthulhu/gen-mtls-certs.sh

@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+# gen-mtls-certs.sh — Generate a private CA and leaf certificates for Gthulhu mTLS.
+#
+# Usage:
+#   ./gen-mtls-certs.sh [OUTPUT_DIR]
+#
+# Environment variables:
+#   DM_EXTRA_SANS  — Extra SANs for the DM server certificate (comma-separated).
+#                    Example: DM_EXTRA_SANS="IP:10.0.0.5,DNS:my-dm.example.com"
+#   HELM_RELEASE   — Helm release name (default: gthulhu). Used to build the
+#                    headless-service wildcard DNS SAN.
+#   NAMESPACE      — Kubernetes namespace (default: default).
+#
+# Outputs (all PEM-encoded):
+#   ca.crt / ca.key           — Private CA
+#   dm.crt / dm.key           — Decision Maker (DM sidecar) server certificate
+#   manager.crt / manager.key — Manager client certificate
+#
+# The DM server cert includes SANs for:
+#   - localhost / 127.0.0.1          (scheduler → sidecar, same pod)
+#   - *.<release>-gthulhu-scheduler-sidecar.<ns>.svc.cluster.local
+#                                     (manager → sidecar, cross-node via headless svc DNS)
+#   - Any extra SANs from DM_EXTRA_SANS
+
+set -euo pipefail
+
+OUT="${1:-certs}"
+mkdir -p "$OUT"
+
+CA_DAYS=3650    # 10 years
+LEAF_DAYS=730   # 2 years
+
+RELEASE="${HELM_RELEASE:-gthulhu}"
+NS="${NAMESPACE:-default}"
+HEADLESS_SVC="${RELEASE}-gthulhu-scheduler-sidecar"
+
+# Build DM SAN string
+DM_SANS="DNS:localhost,IP:127.0.0.1"
+DM_SANS="${DM_SANS},DNS:*.${HEADLESS_SVC}.${NS}.svc.cluster.local"
+DM_SANS="${DM_SANS},DNS:*.${HEADLESS_SVC}.${NS}.svc"
+DM_SANS="${DM_SANS},DNS:${HEADLESS_SVC}.${NS}.svc.cluster.local"
+if [[ -n "${DM_EXTRA_SANS:-}" ]]; then
+  DM_SANS="${DM_SANS},${DM_EXTRA_SANS}"
+fi
+
+echo "==> DM certificate SANs: ${DM_SANS}"
+echo ""
+
+echo "==> Generating private CA …"
+openssl ecparam -name prime256v1 -genkey -noout -out "$OUT/ca.key"
+openssl req -new -x509 -days "$CA_DAYS" \
+  -key "$OUT/ca.key" \
+  -out "$OUT/ca.crt" \
+  -subj "/CN=Gthulhu-Private-CA"
+
+echo "==> Generating DM sidecar server certificate …"
+openssl ecparam -name prime256v1 -genkey -noout -out "$OUT/dm.key"
+openssl req -new \
+  -key "$OUT/dm.key" \
+  -out "$OUT/dm.csr" \
+  -subj "/CN=gthulhu-decisionmaker"
+openssl x509 -req -days "$LEAF_DAYS" \
+  -in "$OUT/dm.csr" \
+  -CA "$OUT/ca.crt" -CAkey "$OUT/ca.key" -CAcreateserial \
+  -extfile <(printf "subjectAltName=${DM_SANS}\nextendedKeyUsage=serverAuth,clientAuth") \
+  -out "$OUT/dm.crt"
+
+echo "==> Generating Manager client certificate …"
+openssl ecparam -name prime256v1 -genkey -noout -out "$OUT/manager.key"
+openssl req -new \
+  -key "$OUT/manager.key" \
+  -out "$OUT/manager.csr" \
+  -subj "/CN=gthulhu-manager"
+openssl x509 -req -days "$LEAF_DAYS" \
+  -in "$OUT/manager.csr" \
+  -CA "$OUT/ca.crt" -CAkey "$OUT/ca.key" -CAcreateserial \
+  -extfile <(printf "extendedKeyUsage=clientAuth") \
+  -out "$OUT/manager.crt"
+
+# Clean up CSRs
+rm -f "$OUT"/*.csr "$OUT"/*.srl
+
+echo ""
+echo "✅  Certificates generated in $OUT/"
+echo ""
+echo "Files:"
+ls -1 "$OUT"
+echo ""
+echo "To install with mTLS enabled:"
+echo "  helm install gthulhu ./gthulhu \\"
+echo "    --set mtls.enabled=true \\"
+echo "    --set-file mtls.ca.cert=$OUT/ca.crt \\"
+echo "    --set-file mtls.dm.cert=$OUT/dm.crt \\"
+echo "    --set-file mtls.dm.key=$OUT/dm.key \\"
+echo "    --set-file mtls.manager.cert=$OUT/manager.crt \\"
+echo "    --set-file mtls.manager.key=$OUT/manager.key"

gthulhu/templates/_helpers.tpl

@@ -78,3 +78,14 @@ MongoDB auth secret name (referencing subchart)
 {{- printf "%s-mongodb-auth" .Release.Name }}
 {{- end }}
 {{- end }}
+
+{{/*
+mTLS Secret name — returns existingSecret if set, otherwise the chart-managed name.
+*/}}
+{{- define "gthulhu.mtlsSecretName" -}}
+{{- if .Values.mtls.existingSecret }}
+{{- .Values.mtls.existingSecret }}
+{{- else }}
+{{- printf "%s-mtls-certs" (include "gthulhu.fullname" .) }}
+{{- end }}
+{{- end }}

gthulhu/templates/configmap.yaml

@@ -6,6 +6,7 @@ metadata:
   labels:
     {{- include "gthulhu.labels" . | nindent 4 }}
     app.kubernetes.io/component: scheduler
+immutable: true
 data:
   jwt_public_key.pem: |
     -----BEGIN PUBLIC KEY-----