feat: validate Keycloak Bearer tokens via FastAPI dependency

[vredchenko]

Summary

Adds Keycloak Bearer-token validation to the FastAPI backend as a global dependency. Every non-exempt request must carry a valid JWT signed by the configured Keycloak realm; the signature is verified offline against the realm's JWKS (fetched once on first request, cached by PyJWKClient for 10 minutes).

Companion PRs:

• Add Keycloak authentication to smartem app smartem-frontend#74 — SPA auth ceremony, attaches the token
• feat: local Keycloak mock + backend auth env wiring smartem-devtools#198 — local Keycloak mock + ConfigMap wiring for KEYCLOAK_* env vars

What's in the change

• src/smartem_backend/auth.py — verify_token FastAPI dependency that:
  • Returns None (no-op) when KEYCLOAK_AUTH_REQUIRED != "true". Tests and existing dev workflows that don't pass tokens stay unaffected.
  • Returns None for exempt paths: /health, /status, /openapi.json, /docs, /redoc.
  • Otherwise extracts the Bearer token, looks up the signing key via PyJWKClient.get_signing_key_from_jwt(token), calls jwt.decode(...) with algorithms=["RS256"], returns claims.
  • Any failure (missing header, wrong scheme, malformed token, bad signature, expired, wrong issuer) → HTTPException(401) with WWW-Authenticate: Bearer.
• src/smartem_backend/api_server.py — wires dependencies=[Depends(verify_token)] at the FastAPI() constructor so it runs ahead of every endpoint.
• pyproject.toml — adds pyjwt[crypto]>=2.9.0,<3.0.0 to the backend extra.
• tests/smartem_backend/test_auth.py — 8 tests covering gated-off default, exempt paths, missing/malformed/empty/non-Bearer Authorization headers, and the config helpers.

All 191 existing backend tests still pass (auth defaults to off, so the global dependency is a no-op under the current test fixtures).

Configuration

Env var	Purpose	Default
KEYCLOAK_AUTH_REQUIRED	Master switch	false
KEYCLOAK_URL	Keycloak base URL	""
KEYCLOAK_REALM	Realm name	dls
KEYCLOAK_CLIENT_ID	Reserved for a future aud/azp check	""
KEYCLOAK_VERIFY_ISS	Enforce iss claim against ${KEYCLOAK_URL}/realms/${KEYCLOAK_REALM}	true

The iss escape hatch exists for the local dev mock, where tokens are minted with the browser-facing URL (localhost:30090) while the pod fetches JWKS via in-cluster DNS (keycloak-service:8080). Signature is sound, but issuer strings don't match — so dev sets KEYCLOAK_VERIFY_ISS=false. Production points at the real DLS realm with a stable hostname and leaves verification on.

ConfigMap defaults for both dev and staging environments live in DiamondLightSource/smartem-devtools#198.

Verified end-to-end

In a local single-node k3s with all three branches applied:

Request	Token	Result
GET /status	none	200
GET /acquisitions	none	401 Missing or malformed Authorization header
GET /acquisitions	malformed bearer	401 Invalid token
GET /acquisitions	real devuser token from local Keycloak mock	200 + payload

Pod-side JWKS reachability against the in-cluster Keycloak service also verified.

Breaking change

When KEYCLOAK_AUTH_REQUIRED=true, every non-exempt endpoint requires a Bearer token. The library default is false so a pure pip install upgrade with no config change is non-breaking, but the staging ConfigMap in DiamondLightSource/smartem-devtools#198 ships with KEYCLOAK_AUTH_REQUIRED=true — flipping the switch in any deployed environment will break any caller that doesn't carry a token.

The agent is not yet token-aware. Issue #284 captures the open question of how SmartEM Agents authenticate against a JWT-protected API (recommended path: Keycloak client_credentials grant with a dedicated SmartEM-agent service-account client). Sequencing the rollout against #284 is a discussion the reviewer should weigh in on — see the release-procedure conversation.

For the version bump on tag — MAJOR is the honest call given the practical impact, even though the library default is off.

Test plan

• pytest tests/smartem_backend/test_auth.py — 8/8 passes
• pytest tests/smartem_backend/ — 192/192 passes (191 existing + 8 new)
• ruff check src/smartem_backend/auth.py src/smartem_backend/api_server.py — clean
• pyright src/smartem_backend/auth.py — 0 errors
• End-to-end smoke test in local k3s with KEYCLOAK_AUTH_REQUIRED=true (table above)
• CI build (RC image) on push to this branch — see workflow run on the latest commit