Federated IPAM: Karmada-based pool delegation to edge clusters

[scotwells]

Overview

Support a federated IPAM topology where IP pools are delegated from a central hub to edge clusters via Karmada, and each edge cluster runs a local IPAM instance that manages allocations autonomously from its delegated pool.

Motivation

In a multi-cluster deployment, edge clusters need to perform IPAM allocations (pod CIDRs, service CIDRs, etc.) locally — without a round-trip to a central control plane for every allocation. At the same time, the overall address space must be centrally coordinated to prevent overlap across clusters.

The IPPool hierarchy introduced in #25 provides the right building blocks. This issue defines how to compose them with Karmada to achieve federated allocation.

Architecture

Two-tier model

Hub (Karmada control plane)

• Holds root IPPool objects covering the full address space
• Operators create child IPPool objects carved from the root, one per edge cluster
• Karmada PropagationPolicy propagates each cluster's child IPPool to that member cluster

Edge (member cluster)

• Receives its delegated IPPool via Karmada — this is its local root
• Runs a local IPAM instance that manages IPClaim and IPAllocation objects against the delegated pool
• May further carve sub-pools (additional child IPPool objects) for per-node, per-namespace, or per-workload-type allocation
• Operates fully autonomously — no round-trip to the hub per allocation

Non-overlapping allocation is guaranteed by construction: the delegated sub-pools are disjoint by the time Karmada propagates them.

Pool lifecycle

Provisioning a cluster:

1. Operator creates a child IPPool on the hub referencing the root pool and a prefix length
2. Hub IPAM allocates the block atomically (SELECT ... FOR UPDATE on parent pool row) and records status.allocatedCIDR
3. Karmada propagates the child IPPool (including status) to the member cluster
4. Local IPAM at the edge treats the propagated pool as its allocation root

Deprovisioning a cluster:

1. Operator deletes the child IPPool on the hub
2. Hub IPAM releases the block back to the root pool
3. Karmada removes the propagated IPPool from the member cluster

Further sub-pool delegation at the edge

Edge clusters can carve sub-pools from their delegated root, using the same IPPool hierarchy:

Hub root pool:       10.0.0.0/8
  └── cluster-a:    10.1.0.0/16   (propagated via Karmada)
        ├── node-1: 10.1.1.0/24   (local child pool, per-node)
        │     └── IPAllocation: 10.1.1.5  (individual pod IP)
        └── node-2: 10.1.2.0/24

No new resource types are required. The existing pool hierarchy handles this natively.

Pool exhaustion

If an edge cluster exhausts its delegated pool, a controller can watch pool utilization (status.capacity.available) and create an additional child IPPool at the hub when capacity drops below a threshold. The new pool is propagated to the edge and made available for further allocation.

IPAM service changes

The IPAM service itself requires no federation-specific changes. It has no awareness of Karmada. The federated topology is an operational composition of existing primitives:

• IPPool hierarchy (from Introduce IPPool, IPClaim, and IPAllocation to replace IPPrefix/IPPrefixClaim #25) provides the address space structure
• Karmada propagation delivers pools to edge clusters
• Local IPAM runs unmodified at each edge

Why status.allocatedCIDR survives propagation

Child pool CIDRs are recorded in status.allocatedCIDR (per the refinements in #25). Karmada must be configured to carry status on propagation for IPPool resources — either via a ResourceInterpreter customization or by propagating the backing IPAllocation object alongside the pool. Without this, the edge IPAM cannot determine the pool's address space.

Karmada configuration (out of scope for this repo)

• PropagationPolicy selecting child IPPool objects by cluster label
• ResourceInterpreter to carry status.allocatedCIDR through propagation
• Controller watching edge pool utilization and creating additional hub-side child pools when needed

Acceptance criteria

• A child IPPool created at the hub and propagated to an edge cluster (with status) is usable as an allocation root by the local IPAM without modification
• IPClaim objects created against a propagated IPPool at the edge produce non-overlapping IPAllocation objects locally
• Edge child IPPool objects (sub-pools of the propagated root) are created and managed by the local IPAM using the same transaction semantics as hub-side pools
• Deleting the hub-side child IPPool releases its CIDR back to the root pool
• No Karmada-specific logic exists in the IPAM service itself

[scotwells]

Telemetry and platform-wide IP utilization

In a federated topology, IP utilization is distributed across hub and edge clusters. This note covers how to maintain a coherent platform-wide view without requiring the hub to track individual edge allocations.

Per-pool utilization via status.capacity

IPPool.status.capacity (total / allocated / available) is the primary utilization signal. Every IPAM instance — hub and edge — maintains this on each pool it manages. This is already part of the #25 design.

For platform-wide rollup, the hub needs status.capacity from delegated pools running at edge clusters. Two mechanisms:

Option A — Karmada aggregated status. Configure a ResourceInterpreter to aggregate status.capacity from member cluster pools back to the hub-side child IPPool. The hub then has utilization for every delegated block without syncing individual IPAllocation objects. This is the preferred approach — it keeps the hub lightweight and avoids fan-out reads.

Option B — Federated metrics scraping. Each edge IPAM exposes Prometheus metrics for pool utilization. A platform-level metrics aggregator (Thanos, federated Prometheus) scrapes all clusters and rolls up utilization globally. More operationally complex but gives richer time-series data for dashboards and alerting.

Both can coexist: Karmada status aggregation for live capacity on the hub-side resource, federated metrics for historical trends and alerting.

Prometheus metrics

Each IPAM instance should expose:

ipam_pool_total_addresses{pool, ip_family}        # total IPs in pool
ipam_pool_allocated_addresses{pool, ip_family}    # currently allocated
ipam_pool_available_addresses{pool, ip_family}    # available
ipam_pool_utilization_ratio{pool, ip_family}      # allocated / total

With a federated metrics layer, these can be aggregated with a cluster label added at scrape time to produce cross-cluster views.

Platform-wide utilization without individual allocation sync

Critically, the hub does not need to know about individual IPAllocation objects at edge clusters to measure utilization. The status.capacity rollup on each delegated IPPool is sufficient:

• Hub root pool status.capacity reflects blocks carved out for clusters (allocated = sum of delegated child pool sizes)
• Each edge pool's status.capacity reflects local allocation within that block
• Platform utilization = edge allocated / hub root total, computable from aggregated status alone

This avoids the fan-out cost of syncing potentially thousands of IPAllocation objects back to the hub.

Pool exhaustion alerting

With federated metrics, alert when ipam_pool_available_addresses drops below a threshold. This triggers the controller (described in the main issue) to request an additional child pool from the hub before the edge cluster exhausts its allocation space.

Additional acceptance criteria

• Each IPAM instance exposes Prometheus metrics for pool utilization (total / allocated / available / utilization ratio) per pool
• Hub-side child IPPool.status.capacity reflects aggregated utilization from the member cluster (via Karmada status aggregation or equivalent)
• Platform-wide IP utilization is computable from hub-side pool status alone, without syncing individual IPAllocation objects from edge clusters

[scotwells]

Implementation Spec: Federated Edge Mode (ipam edge)

This comment tracks the design and implementation plan for adding a stateless, CRD-backed controller-manager mode to the IPAM binary. When running at an edge cluster, the binary starts a controller-runtime manager instead of an aggregated apiserver. No PostgreSQL dependency; no aggregated apiserver wiring.

---

1. Package / file map

cmd/ipam/
  main.go                          ← add NewEdgeCommand() registration (change)
  edge.go                          ← new: "ipam edge" subcommand
internal/
  allocator/
    interface.go                   ← existing PrefixAllocator — hub-only, NOT used at edge
    prefix.go                      ← PostgresPrefixAllocator — hub-only
    kubernetes.go                  ← NEW: KubernetesPrefixAllocator
  allocation/
    cidr.go                        ← shared, zero non-stdlib imports, no change
  controller/
    allocator.go                   ← NEW: narrow PrefixAllocator interface (edge-side)
    ipclaim_controller.go          ← NEW: IPClaim reconciler
    ippool_controller.go           ← NEW: IPPool reconciler
pkg/apis/ipam/v1alpha1/
  types.go                         ← shared types, no change
  register.go                      ← shared scheme registration, no change

Hub-only packages (never imported by edge code):

Package	Why hub-only
internal/apiserver/	aggregated apiserver wiring
internal/registry/ipam/	genericregistry.Store REST handlers
internal/storage/	Postgres RESTOptionsGetter + codec
internal/watch/	Postgres changelog watcher
internal/allocator/interface.go + prefix.go	pgx.Tx-based allocation

Shared with no changes required:

Package	What edge uses
internal/allocation/	FindFirstAvailableBlock, CountAddresses
pkg/apis/ipam/v1alpha1/types.go	IPPool, IPClaim, IPAllocation types
pkg/apis/ipam/v1alpha1/register.go	scheme group/version constants
pkg/apis/ipam/install/	install.Install(scheme)

---

2. New dependency

sigs.k8s.io/controller-runtime v0.19.x  // pin to match k8s.io/* versions

Only new external dependency. Re-exports k8s.io/client-go, k8s.io/apimachinery, k8s.io/api at compatible versions.

---

3. CRD vs aggregated apiserver at the edge

At edge clusters, the IPAM types are present as CRDs — installed by Karmada propagation or a bootstrap step. The controller-manager registers ipamv1alpha1.AddToScheme in its scheme and calls ctrl.NewManager pointing at the edge cluster's kube-apiserver. No aggregated apiserver TLS or delegated auth is needed.

The edge controller-manager is completely unaware of Karmada. Karmada-propagated IPPool objects are just CRD instances in the edge cluster's etcd.

---

4. cmd/ipam/edge.go — the ipam edge subcommand

type EdgeOptions struct {
    Kubeconfig             string
    LeaderElect            bool
    MetricsBindAddress     string
    HealthProbeBindAddress string
}

Flags (no --postgres-dsn):

--kubeconfig                 string   Path to kubeconfig (defaults to in-cluster config)
--leader-elect               bool     Enable leader election (default true)
--metrics-bind-address       string   Metrics endpoint address (default ":8080")
--health-probe-bind-address  string   Health probe address (default ":8081")

RunEdge:

1. Build *rest.Config from --kubeconfig or in-cluster config
2. Construct ctrl.Manager with IPAM scheme registered
3. Construct KubernetesPrefixAllocator backed by the manager's client
4. Register IPClaimReconciler and IPPoolReconciler
5. Call mgr.Start(ctx)

---

5. cmd/ipam/main.go change

Single-line addition:

cmd.AddCommand(NewServeCommand())
cmd.AddCommand(NewMigrateCommand())
cmd.AddCommand(NewVersionCommand())
cmd.AddCommand(NewEdgeCommand())   // ← add

---

6. internal/controller/allocator.go — narrow interface

The existing PrefixAllocator in internal/allocator/ is Postgres-specific (all methods accept pgx.Tx). Define a separate, narrower interface in the controller package:

package controller

type PrefixAllocator interface {
    // AllocatePrefix atomically carves a sub-prefix of prefixLen bits from
    // the pool identified by poolName. The implementation owns its own
    // conflict-resolution loop.
    AllocatePrefix(ctx context.Context, poolName string, prefixLen int, ipFamily string) (net.IPNet, error)

    // ReleasePrefix removes the allocation record for claimName from the pool.
    ReleasePrefix(ctx context.Context, poolName string, claimName string) error
}

This keeps the hub's PrefixAllocator untouched and avoids any pgx import in controller code.

---

7. internal/allocator/kubernetes.go — KubernetesPrefixAllocator

type KubernetesPrefixAllocator struct {
    client   client.Client
    maxRetry int // default 10
}

CAS retry loop (AllocatePrefix)

Leader election (single active reconciler) provides the primary serialisation layer. The CAS loop handles the brief failover window where two leader pods overlap, and concurrent allocations from the IPPool and IPClaim reconcilers within the same manager process.

for attempt := 0; attempt < maxRetry; attempt++:
  1. GET IPPool → pool (with resourceVersion rv)
  2. cidr := pool.Status.AllocatedCIDR  (child pools, set by hub or prior reconcile)
          or pool.Spec.CIDR             (root pools)
  3. LIST IPAllocation where spec.poolRef.name == poolName → existing CIDRs
  4. block, err := allocation.FindFirstAvailableBlock(cidr, existing, prefixLen, strategy)
     ErrPoolExhausted → return ErrPoolExhausted immediately (no retry)
  5. Build IPAllocation with deterministic name (stableAllocationName below)
     Create it; IsAlreadyExists + CIDR matches → idempotent return; IsAlreadyExists otherwise → continue
  6. PATCH pool.Status capacity with resourceVersion rv as precondition
     IsConflict → continue (pool changed under us)
  7. return block, nil
return fmt.Errorf("exhausted %d CAS retries", maxRetry)

Stable allocation names prevent orphaned objects on controller restart:

func stableAllocationName(poolName, claimName string) string {
    h := sha256.Sum256([]byte(poolName + "/" + claimName))
    return "alloc-" + hex.EncodeToString(h[:8])
}

---

8. internal/controller/ipclaim_controller.go

type IPClaimReconciler struct {
    client.Client
    Scheme    *runtime.Scheme
    Allocator PrefixAllocator
}

Reconcile loop:

1. GET IPClaim — not found → return (object gone)
2. DeletionTimestamp set → finalizer path (release allocation, remove finalizer)
3. Ensure finalizer ipam.miloapis.com/allocation present
4. If already bound and IPAllocation exists → return (no-op)
5. Call Allocator.AllocatePrefix
6. Create IPAllocation with ownerRef → IPClaim
7. PATCH claim status: allocatedCIDR, Allocated=True/False condition

Requeue policy:

Scenario	Behavior
Pool not found	Allocated=False, Reason=PoolNotFound, RequeueAfter: 60s
Pool exhausted	Allocated=False, Reason=PoolExhausted, RequeueAfter: 30s
CAS retries exhausted	return error (controller-runtime exponential backoff)
Already bound + allocation present	no-op, return nil

Watches:

ctrl.NewControllerManagedBy(mgr).
    For(&ipamv1alpha1.IPClaim{}).
    Owns(&ipamv1alpha1.IPAllocation{}).  // requeue claim if allocation deleted
    Complete(r)

---

9. internal/controller/ippool_controller.go

Reconciles IPPool objects with spec.parentPoolRef set.

Distinguishing propagated pools from new local child pools: Check status.allocatedCIDR. Non-empty means the CIDR is already authoritative (hub-assigned or prior reconcile) — skip allocation. Empty with spec.parentPoolRef set means a new local child pool needing allocation. No Karmada annotation inspection required.

type IPPoolReconciler struct {
    client.Client
    Scheme    *runtime.Scheme
    Allocator PrefixAllocator
}

Reconcile loop:

1. GET IPPool — not found → return
2. spec.parentPoolRef == nil → root pool, ensure Allocated=True condition, return
3. status.allocatedCIDR != "" → already allocated, ensure conditions correct, return
4. GET parent pool — not found → Allocated=False, Reason=ParentNotFound, RequeueAfter: 30s
5. Call Allocator.AllocatePrefix against parent pool
6. PATCH pool status: allocatedCIDR, capacity fields, Allocated=True/False condition

Watches: Watches all IPPool objects; maps changed pools to their children (pools whose spec.parentPoolRef.name matches) so child pools requeue when their parent becomes ready.

Deletion: Finalizer ipam.miloapis.com/pool-allocation on local child pools. On deletion: release allocation from parent, remove finalizer. Propagated pools that arrive without this finalizer are deleted cleanly without edge action.

---

10. Leader election and serialisation

• --leader-elect=true (default): single active reconciler pod — normal-case single-writer serialisation
• CAS loop: defence against the leader failover window and concurrent allocations from IPPool + IPClaim reconcilers within the same manager goroutine pool
• No cross-pool conflicts: CAS is per-pool; two allocations from different pools cannot conflict

---

11. What is NOT present at the edge

• No pgxpool.Pool — no --postgres-dsn, no Postgres dial
• No genericapiserver.GenericAPIServer — no TLS listener, no auth delegation, no OpenAPI endpoint
• No internal/apiserver/, internal/storage/, internal/watch/, internal/registry/ipam/ imports
• No aggregated apiserver RecommendedOptions

---

12. Implementation checklist

• Add sigs.k8s.io/controller-runtime to go.mod
• cmd/ipam/edge.go — EdgeOptions, NewEdgeCommand, RunEdge
• cmd/ipam/main.go — register NewEdgeCommand()
• internal/controller/allocator.go — narrow PrefixAllocator interface
• internal/allocator/kubernetes.go — KubernetesPrefixAllocator with CAS retry loop
• internal/controller/ipclaim_controller.go — IPClaimReconciler
• internal/controller/ippool_controller.go — IPPoolReconciler
• Unit tests for KubernetesPrefixAllocator using envtest or fake client
• Unit tests for both reconcilers using fake client
• RBAC manifests for edge controller-manager (get/list/watch/patch IPPool, IPClaim; create/delete IPAllocation)
• Taskfile target for edge mode (ipam edge)