Support namespace root policy

[hu-ahmed]

Resolves: #1638

Summary

This PR adds support for namespace root policies in Ditto policy enforcement, including wildcard-based namespace mappings.

A namespace can be mapped to one or more root policy IDs. During enforcer creation, Ditto transparently merges entries from those root policies into policies of that namespace.

What changed

• Added namespace-root merge support in policy enforcer resolution.
• Added config abstraction:
  • NamespacePoliciesConfig
  • DefaultNamespacePoliciesConfig
• Added support for namespace policy patterns:
  • exact namespace: org.example.devices
  • prefix wildcard: org.example.devices.*
  • catch-all: *
• Added deterministic precedence for overlapping patterns:
  • exact namespace
  • more specific prefix wildcard
  • broader prefix wildcard
  • catch-all *
• Wired namespace policy resolution into:
  • cache loader path
  • create-policy enforcement path (PolicyEnforcerActor)
• Extended cache invalidation:
  • when a namespace root policy changes, cached policies in covered namespaces are invalidated
• Added canonical base config file:
  • internal/utils/config/.../ditto-namespace-policies.conf
  • included via ditto-service-base.conf
• Updated Helm templates and values to service-scoped config:
  • policies.config.namespacePolicies
  • things.config.namespacePolicies
• Updated chart docs accordingly.
• Added config validation for unsupported wildcard syntax at startup.

Behavior / rules

• Only entries with importable = "implicit" are merged.
• Entries with importable = "explicit" or importable = "never" are not merged.
• Local policy entries win on label conflicts.
• If multiple namespace root policies match, they are applied in deterministic precedence order:
  1. exact namespace
  2. more specific wildcard prefix
  3. broader wildcard prefix
  4. *
• If a configured root policy is missing or deleted, entries are skipped and an error is logged.
• Stored policy JSON is not modified; merging happens only at enforcer-build time.
• Unsupported namespace policy patterns are rejected at config load time.

Supported config syntax

• org.example.devices
• org.example.devices.*
• *

Unsupported examples:

• org.*.devices
• foo*
• **

Example config

policies:
  config:
    namespacePolicies:
      org.example.devices:
        - org.example:tenant-root-exact
      org.example.devices.*:
        - org.example:tenant-root-devices
      org.example.*:
        - org.example:tenant-root-general

things:
  config:
    namespacePolicies:
      org.example.devices:
        - org.example:tenant-root-exact
      org.example.devices.*:
        - org.example:tenant-root-devices
      org.example.*:
        - org.example:tenant-root-general

[thjaeckle]

QA Review — Namespace Root Policies

Nice work overall — the core design is clean (runtime-only merge, stored policies untouched, fully opt-in). The precedence model, config validation, and cache invalidation extension are well thought through.

Two findings I'd like to flag at the PR level:

---

CRITICAL: Things-Search service is not updated

thingsearch/service's ResolvedPolicyCacheLoader also calls policy.withResolvedImports() when building search index entries. If namespace root policy entries grant READ on thing resources, those permissions must also be visible in search results.

Without this, search results will be inconsistent with actual enforcement — a user granted access via a namespace root policy can access a thing via direct API but will not find it via search.

This is likely the highest-impact gap in the current PR scope.

---

HIGH: No protection against self-referential namespace root policies

If org.example:tenant-root is in namespace org.example and the config has:

"org.example.*" = ["org.example:tenant-root"]

then when the cache loads org.example:tenant-root itself:

1. PolicyEnforcerCacheLoader.asyncLoad() loads the raw policy
2. withResolvedImportsAndNamespacePolicies() calls mergeNamespacePolicies()
3. getRootPoliciesForNamespace("org.example") returns [org.example:tenant-root]
4. The resolver tries to load org.example:tenant-root again via PolicyCacheLoader

The raw PolicyCacheLoader is a separate loader, so this won't deadlock on the enforcer cache. However, the root policy ends up attempting to merge its own implicit entries into itself — a no-op due to label dedup, but wasted work and a confusing LOG.error if the policy happens to be missing during its own initial load.

Suggested guard in mergeNamespacePolicies:

if (namespacePoliciesConfig.getAllNamespaceRootPolicyIds()
        .contains(resolvedPolicy.getEntityId().orElse(null))) {
    return CompletableFuture.completedFuture(resolvedPolicy);
}

[thjaeckle]

I also noticed that documentation of this new feature is missing

[thjaeckle]

Review Round 2 — Namespace Root Policies

Good progress since round 1 — the search service integration, per-service config includes, documentation, PolicyEnforcerActor config-sharing fix, parallel root policy resolution, and transitive cache invalidation all look solid.

---

CRITICAL: BackgroundSyncStream not updated — perpetual re-indexing

ResolvedPolicyCacheLoader now adds namespace root policy tags to referencedPolicies, which end up in the search index as allReferencedPolicyTags. But BackgroundSyncStream.checkReferencedPoliciesForConsistency() still compares:

final List<PolicyId> importedPolicyIds = optionalPolicy.get().getPolicyImports()
        .stream().map(PolicyImport::getImportedPolicyId).toList();
// ...
if (importedPolicyIds.size() != (indexedReferencedPolicyTags.size() - 1)) {
    return CompletableFuture.completedFuture(false); // triggers re-index
}

importedPolicyIds counts only explicit policy imports from the stored policy, but the indexed allReferencedPolicyTags now includes namespace root policy tags too. For any policy in a namespace with root policies, this size check will always fail, triggering constant re-indexing on every background sync cycle.

This will cause significant unnecessary MongoDB load proportional to the number of things in affected namespaces.

Fix: Update checkReferencedPoliciesForConsistency to also account for namespace root policy tags when computing the expected referenced policies count, or switch to a set-equality check.

---

HIGH: Self-referential guard too broad in both PolicyEnforcer and ResolvedPolicyCacheLoader

Both paths now have the same guard:

if (entityId.isPresent() && namespacePoliciesConfig.getAllNamespaceRootPolicyIds().contains(entityId.get())) {
    return CompletableFuture.completedFuture(resolvedPolicy);
}

This skips all namespace root merging for any policy that appears in the root policy set — not just self-references. Example:

• Config: "com.acme.*" = ["com.acme:acme-root"], "*" = ["global:catch-all"]
• When building the enforcer for com.acme:acme-root, the guard fires (it's in getAllNamespaceRootPolicyIds()) and skips ALL merging — so global:catch-all entries are never applied, even though they should be.

The guard correctly prevents deadlocks/wasted work for self-reference, but the fix should be surgical — filter out only the self-reference:

final List<PolicyId> rootPolicies = namespacePoliciesConfig.getRootPoliciesForNamespace(namespace)
        .stream()
        .filter(rootId -> !entityId.map(rootId::equals).orElse(false))
        .toList();

This preserves cross-namespace root policy merging while still preventing self-reference.

---

HIGH: Helm config divergence risk — three independent values for one semantic config

The Helm chart defines namespace root policies as three separate, independent values:

Template	Reads from
policies-extension.conf.tpl	.Values.policies.config.namespacePolicies
things-extension.conf.tpl	.Values.things.config.namespacePolicies
search-extension.conf.tpl	.Values.thingsSearch.config.namespacePolicies

There is no mechanism to keep them in sync. If they diverge, enforcement becomes inconsistent: a subject granted READ via a namespace root policy in the policies service might not see the thing in search results, or vice versa. This is always a bug, never intentional.

The chart already has a global: section for cross-service config (prometheus, logging, tracing, cluster). Namespace root policies are a textbook case for this pattern.

Suggested fix: Define a single global.namespacePolicies: {} in values.yaml and have all three templates reference .Values.global.namespacePolicies.

---

MEDIUM: Documentation Helm example uses YAML | (literal block scalar)

In installation-operating.md, the Helm configuration example shows:

policies:
  config:
    namespacePolicies: |
      "org.eclipse.ditto.*" = ["org.eclipse.ditto:tenant-root"]

The | syntax renders namespacePolicies as a string, not a YAML map. But values.yaml defines namespacePolicies: {} (a map) and the Helm templates iterate over it with range. Passing a string will cause a Go template error at deploy time.

Should be:

policies:
  config:
    namespacePolicies:
      "org.eclipse.ditto.*":
        - "org.eclipse.ditto:tenant-root"

(This also applies once the Helm values are consolidated under global per the finding above.)

---

MEDIUM: Config validation order in DefaultNamespacePoliciesConfig.of()

patternPrecedence(namespace) is called before value.valueType() != ConfigValueType.LIST. A non-list entry with an invalid namespace pattern (e.g. "org.*.devices" = "not-a-list") throws DittoConfigError, even though non-list values are silently skipped on the next line.

Either move validation after the type check, or (better) also explicitly reject non-list values with a clear error for fail-fast behavior.

---

Summary

Severity	Count	Key issues
CRITICAL	1	BackgroundSyncStream perpetual re-indexing
HIGH	2	Self-referential guard too broad; Helm config divergence risk
MEDIUM	2	Documentation Helm YAML syntax; config validation order

[thjaeckle]

System tests passed: https://github.com/eclipse-ditto/ditto/actions/runs/23426733200

[thjaeckle]

Hey @hu-ahmed

Thanks a lot for another great contribution.
I now also finished with my manual test and FMPOV this is good to merge.

Please squash into a single commit to get rid of the "resolve comments" commits in the history. After that, it's good to be merged 👍