Temporal Worker Controller demo

A hands-on demo of Worker Versioning + the Temporal Worker Controller on Kubernetes. A small FastAPI service starts test workflows and reads worker-deployment status from the cluster. A React UI runs four scenarios and shows the rollout state in real time.

You'll see how four common workflow shapes behave during a rolling upgrade from worker version A to version B:

• A — a long pinned workflow stays on its starting version
• B — an auto-upgrade workflow can finish on a newer version
• C — a workflow type missing on B fails until you roll back to A
• D — a pinned workflow that uses continue_as_new to hand off to a newer version

You will need

Tool	Purpose
Docker (Rancher Desktop recommended)	Build worker-controller-demo:v-a / :v-b images
Kubernetes (Rancher Desktop, kind, k3d, etc.)	Runs the controller + worker pods
kubectl	Apply manifests, watch state
Helm 3	Install the worker controller
Temporal Cloud account (or self-hosted ≥ 1.29.1)	Worker Versioning enabled
uv (Python) and Node.js + npm	Run the demo API and UI on your laptop

Setup (one-time)

1. Install the worker controller

# Worker controller CRDs + chart (pick a release from
# https://github.com/temporalio/temporal-worker-controller/releases)
helm install temporal-worker-controller-crds \
  oci://docker.io/temporalio/temporal-worker-controller-crds \
  --version <VERSION> --namespace temporal-system --create-namespace

helm install temporal-worker-controller \
  oci://docker.io/temporalio/temporal-worker-controller \
  --version <VERSION> --namespace temporal-system

# Sanity check — both controller pods should be Running
kubectl get pods -n temporal-system

cert-manager? Not needed for this demo. The controller's optional admission webhook (used by WorkerResourceTemplate) is the only thing that wants TLS, and we don't use it. If a future upgrade prompts a webhook error, install cert-manager then.

2. Create the demo namespace and your Temporal Cloud API key Secret

In Temporal Cloud, create an API key (UI → API Keys → Create). Then:

kubectl create namespace worker-controller-demo
kubectl create secret generic temporal-api-key -n worker-controller-demo \
  --from-literal=api-key='YOUR_API_KEY'

3. Apply the TemporalConnection (tells the controller how to reach Temporal)

cp k8s/temporal-connection.example.yaml k8s/temporal-connection.yaml
# Edit spec.hostPort to your regional gRPC host (e.g. us-east-1.aws.api.temporal.io:7233)
kubectl apply -f k8s/temporal-connection.yaml

4. Build the two worker images

# v-a: registers all workflow types
docker build -t worker-controller-demo:v-a --build-arg DEMO_WORKER_VERSION=a .

# v-b: omits ONLY RollbackWorkflow (Scenario C will fail on v-b until rollback).
# RolloutGate stays registered so the controller's rollout gate succeeds and the ramp completes.
docker build -t worker-controller-demo:v-b --build-arg DEMO_WORKER_VERSION=b \
  --build-arg DEMO_OMIT_ROLLBACK=1 .

5. Apply the TemporalWorkerDeployment (starts v-a workers)

cp k8s/temporal-worker-deployment.example.yaml k8s/temporal-worker-deployment.yaml
# Edit spec.workerOptions.temporalNamespace to your Temporal Cloud namespace
kubectl apply -f k8s/temporal-worker-deployment.yaml

# Watch until CURRENT and TARGET are both v-a-<hash> and RolloutComplete
kubectl get twd -n worker-controller-demo -w

6. Fill in .env for the demo API

cp .env.example .env

Variable	Value
TEMPORAL_ADDRESS	Your regional gRPC host (matches the TemporalConnection)
TEMPORAL_NAMESPACE	Same as spec.workerOptions.temporalNamespace in the TWD
TEMPORAL_API_KEY	The API key value you put in the K8s Secret
TEMPORAL_TASK_QUEUE	worker-controller-demo (matches the TWD)
K8S_NAMESPACE	worker-controller-demo
K8S_TWD_NAME	worker-controller-demo
TEMPORAL_DEPLOYMENT_NAME	worker-controller-demo/worker-controller-demo (set this — controller prefixes the K8s namespace; needed for pin-to-current in Scenario A)

Keep .env out of git (it's already in .gitignore).

Run the demo

1. Start the API and UI

uv sync
uv run demo-api                 # terminal 1

cd web && npm install
npm run dev                      # terminal 2

Open http://localhost:5173. The top panel shows live TemporalWorkerDeployment status. Four cards below run the scenarios.

2. Verify steady state on v-a

kubectl get twd -n worker-controller-demo should show CURRENT = TARGET = v-a-<hash>, RolloutComplete.

3. Start Scenarios A and B on v-a

In the UI, click Run scenario A, then Run scenario B. Both kick off on v-a:

• A (pinned-demo-…): pinned to v-a; probes, sleeps ~90s, probes again.
• B (auto-demo-…): auto-upgrade; probes (ok-a), sleeps 150s, probes again.

Keep them running. They give the rollout something interesting to do.

4. Roll forward to v-b and watch the ramp

# Edit k8s/temporal-worker-deployment.yaml:
#   image: worker-controller-demo:v-b
#   DEMO_WORKER_VERSION: "b"
kubectl apply -f k8s/temporal-worker-deployment.yaml
kubectl get twd -n worker-controller-demo -w

The controller starts a RolloutGate workflow on v-b → it succeeds (v-b registers RolloutGate) → ramp progresses 25% → 50% → 75% → Current. While that's happening:

• A stays pinned to v-a and completes there (pinned workflows never move).
• B finishes its second probe on whichever build is Current at that moment. If the ramp finishes before B wakes up, result is ok-a -> ok-b; otherwise ok-a -> ok-a.

5. Start Scenarios C and D on v-b

Once CURRENT = v-b-<hash>, click Run scenario C, then Run scenario D:

• C (rollback-demo-…): auto-upgrade; dispatched to v-b. v-b doesn't register RollbackWorkflow → workflow task fails repeatedly with class not registered. The workflow stays Running (no timeout) and keeps retrying.
• D (can-demo-…): pinned to v-b; probes (ok-b), sleeps 2 minutes 30 seconds, then calls continue_as_new with initial_versioning_behavior=AUTO_UPGRADE.

6. Roll back to v-a (while D is still sleeping)

# Edit k8s/temporal-worker-deployment.yaml:
#   image: worker-controller-demo:v-a
#   DEMO_WORKER_VERSION: "a"
kubectl apply -f k8s/temporal-worker-deployment.yaml

The controller ramps back to v-a as Current.

7. Observe C and D both complete on v-a

• C recovers: its next workflow-task retry is auto-upgraded to Current = v-a → v-a registers RollbackWorkflow → runs the activity → returns ok-a → Completed. The workflow was never lost.
• D hands off: when its 2:30 timer fires, gen 0 closes on v-b (ContinuedAsNew); gen 1 starts with AUTO_UPGRADE and lands on Current = v-a → probes → returns gen=1 probe=ok-a → Completed. A pinned workflow safely moved from v-b to v-a at the CaN boundary.

You've now seen all four behaviors in one continuous flow: pinned (A) stays put, auto-upgrade (B) moves, a missing workflow type on v-b (C) blocks until rollback then recovers, and continue-as-new with AUTO_UPGRADE (D) hands off to whichever build is Current at the boundary.

Reset when things get stuck

If a ramp halts or you want a clean baseline:

kubectl delete twd worker-controller-demo -n worker-controller-demo
# Wait for pods to drain
kubectl get pods -n worker-controller-demo

# Flip k8s/temporal-worker-deployment.yaml back to image: worker-controller-demo:v-a
kubectl apply -f k8s/temporal-worker-deployment.yaml

Drained worker-deployment versions on the Temporal side are harmless leftover bookkeeping; they don't block a fresh apply unless you reuse the exact same pod template hash. Bumping the image tag (e.g. :v-b1) is the simplest way to force a brand-new build id.

Common questions

• The UI shows "pin skipped" for Scenario A or C. Set TEMPORAL_DEPLOYMENT_NAME=<k8s-namespace>/<twd-name> in .env (the controller prefixes the K8s namespace; the API needs the full name to pin to the right (deployment, build) pair).
• Status panel stays empty. demo-api uses your kubeconfig to read TWD status. Run kubectl get twd -n worker-controller-demo from the same machine to confirm the context is right.
• Self-hosted Temporal. Set TemporalConnection.spec.hostPort to your frontend (e.g. temporal-frontend.temporal:7233) and follow the controller's configuration guide for TLS/mTLS. Drop TEMPORAL_API_KEY from worker pods if unused.

Repository layout

• activity/ — probe_version, slow_step
• workflows/ — PinnedDemo (A), AutoUpgradeDemo (B), RollbackWorkflow + RolloutGate (C), PinnedCanDemo (D)
• worker/ — versioned worker with readiness probe on :8080
• api/ — FastAPI service (demo-api)
• web/ — Vite + React UI (proxies /api to the API)
• k8s/ — example TemporalConnection and TemporalWorkerDeployment manifests

References

• Worker Versioning
• Temporal Worker Controller
• Continue-as-new