Infrastructure connection configs should be bootstrap-only, not mutable at runtime

[ferr3ira-gabriel]

Problem

The systemplane currently allows runtime mutation of infrastructure connection settings (postgres.*, redis.*, rabbitmq.*) via the /v1/system/configs API. This creates a critical chicken-and-egg problem that can permanently brick the application.

The Circular Dependency

Pod starts → Needs PostgreSQL connection → Reads config from systemplane →
Systemplane is stored IN PostgreSQL → But postgres.primary_host has wrong value →
Cannot connect to PostgreSQL → Pod crashes → No way to fix the configuration

Since the systemplane stores its configuration in the same PostgreSQL database it is trying to configure, if someone changes postgres.primary_host via the API to an incorrect value:

1. The current pod may continue working (existing connection)
2. All new pods will fail to start - they cannot reach the systemplane store
3. The configuration cannot be fixed via API - the API is unreachable
4. Manual database intervention is required to recover

Affected Keys

The following keys are currently marked as MutableAtRuntime: true but should be false:

PostgreSQL (self-locking - systemplane lives here):

• postgres.primary_host
• postgres.primary_port
• postgres.primary_user
• postgres.primary_db
• postgres.replica_host
• postgres.replica_port
• postgres.replica_user
• postgres.replica_db

Redis (often used for caching/sessions):

• redis.host
• redis.master_name

RabbitMQ (core messaging infrastructure):

• rabbitmq.host
• rabbitmq.port
• rabbitmq.user

Additional Issue: Configuration Sharing Between Pods

Because systemplane persists non-default values to PostgreSQL via SeedStore(), these values become shared across all pods. If the first pod resolves a DNS hostname to an IP address and stores it, or if infrastructure changes, new pods receive stale values from the store instead of fresh values from environment variables.

This was observed in production when:

1. Old pods were healthy with existing connections
2. New pods (after restart) failed to connect
3. Investigation revealed pods were using IP addresses (21.26.7.96) from the systemplane store instead of DNS hostnames (postgresql.dev.firmino.lerian.net) from environment variables

Proposed Solution

Change infrastructure connection keys to be bootstrap-only:

{
    Key:              "postgres.primary_host",
    Kind:             domain.KindConfig,
    AllowedScopes:    []domain.Scope{domain.ScopeGlobal},
    DefaultValue:     defaultPGHost,
    ValueType:        domain.ValueTypeString,
    Validator:        validateNonEmptyString,
    ApplyBehavior:    domain.ApplyBootstrapOnly,  // <- Change from ApplyBundleRebuild
    MutableAtRuntime: false,                       // <- Change from true
    Description:      "PostgreSQL primary host address",
    // ...
}

What Should Remain Mutable

Settings that are safe for runtime tuning should stay mutable:

• Connection pool sizes (postgres.max_open_conns, redis.pool_size)
• Timeouts (postgres.query_timeout_sec, redis.read_timeout_ms)
• Application-level settings (rate limits, feature flags, etc.)

Migration Consideration

Existing deployments may have these values stored in the systemplane table. A migration or documentation should be provided to:

1. Remove stored infrastructure host/port values from the systemplane table
2. Ensure environment variables are the sole source of truth for these settings

Impact

• Severity: High - can cause permanent application lockout
• Affected versions: All versions using systemplane with current key definitions
• Workaround: Manual PostgreSQL intervention to delete/fix systemplane entries

[ferr3ira-gabriel]

Additional Concerns

Configuration Loss When Changing Database Host

Another critical issue: if someone changes postgres.primary_host via the systemplane API to point to a different database, the application will:

1. Connect to the new database successfully (assuming it exists)
2. Lose access to ALL previously stored configuration - the systemplane data lives in the old database
3. Start with default values or empty configuration
4. Have no way to recover the previous settings without manual intervention

This is essentially a silent data loss scenario - the application appears to work but has lost its entire configuration history.

Real-World Case: Firmino Cluster DNS Migration

We are currently experiencing this exact problem in the Firmino production cluster while migrating from an old PostgreSQL DNS record to a new one:

• Old DNS: postgresql.dev.lerian.net
• New DNS: postgresql.dev.firmino.lerian.net

What happened:

1. Old matcher pods were running with connections to the old DNS hostname
2. The systemplane seeded the old hostname into the PostgreSQL store
3. We updated the Kubernetes environment variables to use the new DNS hostname
4. When pods restarted, they read the old hostname from systemplane instead of the new environment variable
5. New pods failed to connect because the old DNS no longer resolves correctly
6. Old pods continue working (existing connections), but the cluster cannot scale or recover from pod crashes

The fix requires:

1. Manual database access to delete/update systemplane entries
2. Or keeping the old DNS record alive indefinitely
3. Or a coordinated "big bang" restart that somehow bypasses systemplane

This situation would have been completely avoided if infrastructure hosts were bootstrap-only and always read from environment variables.