chore(rel): add postgres collation mismatch to docs (#1168)

jdpleiness · web-flow · commit 98cae7d89bda · 2025-06-03T09:31:43.000-04:00
Add docs for Postgres collation mismatch known issue.

Docs include release notes warnings and links to resolutions page as well as the resolutions page itself.
diff --git a/docs/admin/postgresql_collation_version_mismatch_resolution.mdx b/docs/admin/postgresql_collation_version_mismatch_resolution.mdx
@@ -0,0 +1,252 @@
+# PostgreSQL Collation Version Mismatch Resolution
+
+# Issue Summary
+
+Customers whom initially deployed Sourcegraph versions prior to v6.2.2553 using the Sourcegraph provided PostgreSQL containers may encounter PostgreSQL collation version mismatch warnings after upgrading to more recent Sourcegraph versions due to an underlying glibc version update.
+
+# Impact
+
+Mismatched collation versions can lead to database index corruption if left unchecked.
+
+# Affected Services
+
+* `pgsql` container
+* `codeintel-db` container
+* `codeinsights-db` container
+
+Only self-hosted customers using the Sourcegraph provided PostgreSQL container images are affected.
+
+Self-hosted customers using external databases, such as AWS RDS, GCP CloudSQL, or another self-managed solution are **NOT** affected.
+
+Sourcegraph Cloud customers are not affected.
+
+# Symptoms
+
+When logging into the database via `psql` or similar tools you may see the following warning:
+
+```shell
+WARNING: database "sg" has a collation version mismatch DETAIL: The database was created using collation version 2.40, but the operating system provides version 2.41.
+```
+
+# Backup Instructions (Required Before Proceeding)
+
+Before performing any reindexing operations, it is critical to back up your data to prevent potential data loss. Use the method applicable to your deployment and your current environment.
+
+Docker example:
+
+```
+docker exec -t <database-container-name> pg_dumpall -U sg > backup_<container>.sql
+```
+
+# Resolution Steps
+
+Each affected database requires a reindex to be performed. This requires downtime, and should be done in the next available maintenance window.
+
+Before reindexing, check the current collation versions. Run the following SQL to compare current and OS collation versions:
+
+```
+SELECT
+    datname,
+    datcollversion,
+    pg_database_collation_actual_version(oid) AS current_os_version
+FROM pg_database;
+```
+
+`datcollversion` should not match `current_os_version`
+
+## Docker Compose deployments
+
+1. Stop all Sourcegraph containers
+
+```
+docker compose down
+```
+
+2. Start only the database containers
+
+```
+docker compose up pgsql codeinsights-db codeintel-db -d
+```
+
+3. Access each database container
+
+```
+docker exec -it <database-container-name> bash
+```
+
+4. Reindex the databases in the container
+
+```
+reindexdb --no-password --username sg --verbose --echo --system
+reindexdb --no-password --username sg --verbose --echo --all
+```
+
+5. Refresh the collation version for each database
+
+```
+psql --no-password --username "sg" --tuples-only -c "SELECT 'ALTER DATABASE ' || datname || ' REFRESH COLLATION VERSION;' FROM pg_database WHERE datname != 'template0';" | psql --no-password --username "sg"
+```
+
+\* For the above commands, replace `sg` with your custom username if one was configured in your environment.
+\*\* For the `codeinsights-db` container, the default username is `postgres` instead of `sg`.
+
+Repeat steps 3 through 5 with all containers:
+
+* `pgsql` container
+* `codeintel-db` container
+* `codeinsights-db` container
+
+Once this is complete, you may start your Sourcegraph deployment
+
+```
+docker compose up -d
+```
+
+## Helm Deployments
+
+1. Stop all non-database Sourcegraph pods (assuming a namespace of `sourcegraph`)
+
+```
+kubectl get -n sourcegraph deploy --no-headers | awk '{print $1}' | xargs -n 1 -P 8 -I % kubectl -n sourcegraph scale deployment % --replicas=0
+```
+
+```
+kubectl -n sourcegraph get sts --selector 'app.kubernetes.io/component!=codeinsights-db,app.kubernetes.io/component!=codeintel-db,app.kubernetes.io/component!=pgsql' --no-headers | awk '{print $1}' | xargs -n 1 -P 8 -I % kubectl -n sourcegraph scale sts % --replicas=0
+```
+
+2. Access each database pod
+
+```
+kubectl exec -it <database-pod-name> -n sourcegraph -- /bin/sh
+```
+
+3. Reindex the databases in the container
+
+```
+reindexdb --no-password --username sg --verbose --echo --system
+reindexdb --no-password --username sg --verbose --echo --all
+```
+
+4. Refresh the collation version for each database
+
+```
+psql --no-password --username "sg" --tuples-only -c "SELECT 'ALTER DATABASE ' || datname || ' REFRESH COLLATION VERSION;' FROM pg_database WHERE datname != 'template0';" | psql --no-password --username "sg"
+```
+
+\* For the above commands, replace `sg` with your custom username if one was configured in your environment.
+\*\* For the `codeinsights-db` container, the default username is `postgres` instead of `sg`.
+
+Repeat steps 2 through 4 with all pods:
+
+* `pgsql-0` pod
+* `codeintel-db-0` pod
+* `codeinsights-db-0` pod
+
+Once this is complete, you may start your Sourcegraph deployment, for example
+
+```
+helm upgrade -n sourcegraph <your-helm-charts>
+```
+
+## Kustomize Deployments
+
+1. Stop all non-database Sourcegraph pods (assuming a namespace of `sourcegraph`). In your cluster Kustomization file (`instances/my-sourcegraph/kustomize.yaml`), uncomment the [multi-version-upgrade util](https://github.com/sourcegraph/deploy-sourcegraph-k8s/blob/main/instances/template/kustomization.template.yaml#L155). This will scale down all non-database deployments and statefulSets replicas to 0\.
+
+```
+  # - ../../components/utils/uid # -- Run all Postgres database with valid users on host
+  - ../../components/utils/multi-version-upgrade # -- Scale down non-database pods to 0 for multi-version upgrade
+  # - ../../components/utils/migrate-to-nonprivileged # -- Component for migrating from privileged to non-privileged
+  #
+```
+
+2. Generate and apply a new \`cluster.yaml\` file
+
+```
+kubectl kustomize instances/my-sourcegraph -o cluster.yaml
+kubectl apply --prune -l deploy=sourcegraph -f cluster.yaml
+```
+
+3. Access each database pod
+
+```
+kubectl exec -it <database-pod-name> -n sourcegraph -- /bin/sh
+```
+
+4. Reindex the databases in the container
+
+```
+reindexdb --no-password --username sg --verbose --echo --system
+reindexdb --no-password --username sg --verbose --echo --all
+```
+
+5. Refresh the collation version for each database
+
+```
+psql --no-password --username "sg" --tuples-only -c "SELECT 'ALTER DATABASE ' || datname || ' REFRESH COLLATION VERSION;' FROM pg_database WHERE datname != 'template0';" | psql --no-password --username "sg"
+```
+
+\* For the above commands, replace `sg` with your custom username if one was configured in your environment.
+\*\* For the `codeinsights-db` container, the default username is `postgres` instead of `sg`.
+
+Repeat steps 3 through 5 with all pods:
+
+* `pgsql-0` pod
+* `codeintel-db-0` pod
+* `codeinsights-db-0` pod
+
+Once this is complete, you may start your Sourcegraph deployment.
+
+Comment out the [multi-version-upgrade util](https://github.com/sourcegraph/deploy-sourcegraph-k8s/blob/main/instances/template/kustomization.template.yaml#L155) in your cluster Kustomization file (`instances/my-sourcegraph/kustomize.yaml`).
+
+```
+# - ../../components/utils/uid # -- Run all Postgres database with valid users on host
+# - ../../components/utils/multi-version-upgrade # -- Scale down non-database pods to 0 for multi-version upgrade
+# - ../../components/utils/migrate-to-nonprivileged # -- Component for migrating from privileged to non-privileged
+```
+
+Generate and apply a new \`cluster.yaml\` file
+
+```
+kubectl kustomize instances/my-sourcegraph -o cluster.yaml
+kubectl apply --prune -l deploy=sourcegraph -f cluster.yaml
+```
+
+## AMI/Machine Image based deployments
+
+Our AMI and GCP machine image deployments run Kubernetes internally, and thus follow the same general directions as our Helm Deployments described above.
+
+To access the deployment you must SSH into your AMI/Machine image first.
+
+* For AWS AMIs, the default username is `ec2-user`
+* For GCP machine images, the default username is `sourcegraph`
+
+Follow the directions laid out in the Helm Deployments section.
+
+# Expected Downtime
+
+* Reindexing may take 15-60 minutes in most cases, but depends on the database size, disk speed, and available CPU/RAM
+* For customers with larger databases \- greater than 100Gb \- or low resource allocation, reindexing may take longer
+
+# Verification
+
+Check that warnings no longer appear when connecting to the database:
+
+```
+psql --no-password --username "$PGUSER" -d sg
+```
+
+After Reindexing: Verify Collation Version:
+
+```
+SELECT
+    datname,
+    datcollversion,
+    pg_database_collation_actual_version(oid) AS current_os_version
+FROM pg_database;
+```
+
+Ensure `datcollversion` matches `current_os_version` for all databases listed.
+
+# Questions?
+
+Contact Sourcegraph Support ([support@sourcegraph.com](mailto:support@sourcegraph.com)) if you encounter issues during this process.
diff --git a/docs/admin/updates/docker_compose.mdx b/docs/admin/updates/docker_compose.mdx
@@ -11,6 +11,31 @@ For upgrade procedures or general info about sourcegraph versioning see the link
 >
 > ***If the notes indicate a patch release exists, target the highest one.***
 
+## v6.2.2553
+
+### Known issues
+
+Customers running Sourcegraph versions prior to v6.2.2553 and using the Sourcegraph provided PostgreSQL containers may encounter PostgreSQL collation version mismatch warnings after upgrading to more recent Sourcegraph versions due to an underlying glibc version update.
+
+When logging into the database via psql or similar tools you may see the following warning:
+
+```shell
+WARNING: database "sg" has a collation version mismatch DETAIL: The database was created using collation version 2.40, but the operating system provides version 2.41.
+```
+
+Mismatched collation versions can lead to database index corruption if left unchecked.
+
+Affected Services
+- pgsql container
+- codeintel-db container
+- codeinsights-db container
+
+Only self-hosted customers using the Sourcegraph provided PostgreSQL container images are affected.
+
+Self-hosted customers using external databases, such as AWS RDS, GCP CloudSQL, or another self-managed solution are NOT affected.
+
+See our [PostgreSQL Collation Version Mismatch Resolution](/admin/postgresql_collation_version_mismatch_resolution) notes for more details.
+
 ## v6.0.0
 
 - Sourcegraph 6.0.0 no longer supports PostgreSQL 12, admins must upgrade to PostgreSQL 16. See our [postgres 12 end of life](/admin/postgres12_end_of_life_notice) notice! As well as [supporting documentation](/admin/postgres) and advisements on how to upgrade.
diff --git a/docs/admin/updates/index.mdx b/docs/admin/updates/index.mdx
@@ -126,6 +126,7 @@ If your instance has schema drift or unfinished oob migrations you may need to a
 - [**Sourcegraph AWS AMI instances**](/admin/deploy/machine-images/aws-ami#upgrade)
 
 ## Other helpful links
+- [PostgreSQL Collation Version Mismatch Resolution](/admin/postgresql_collation_version_mismatch_resolution)
 - [Postgres 12 End Of Life Notice](/admin/postgres12_end_of_life_notice)
 - [Migrator operations](/admin/updates/migrator/migrator-operations)
 - [Upgrading Early Versions](/admin/updates/migrator/upgrading-early-versions)
diff --git a/docs/admin/updates/kubernetes.mdx b/docs/admin/updates/kubernetes.mdx
@@ -17,6 +17,31 @@ For upgrade procedures or general info about sourcegraph versioning see the link
 - The repo-updater service is no longer needed and will be removed from deployment methods going forward.
 - The symbols service and searcher service have been merged and symbols will be removed from deployment methods going forward. Consider moving env vars set on symbols to the searcher deployment before upgrading and reallocating resources from symbols to searcher.
 
+## v6.2.2553
+
+### Known issues
+
+Customers running Sourcegraph versions prior to v6.2.2553 and using the Sourcegraph provided PostgreSQL containers may encounter PostgreSQL collation version mismatch warnings after upgrading to more recent Sourcegraph versions due to an underlying glibc version update.
+
+When logging into the database via psql or similar tools you may see the following warning:
+
+```shell
+WARNING: database "sg" has a collation version mismatch DETAIL: The database was created using collation version 2.40, but the operating system provides version 2.41.
+```
+
+Mismatched collation versions can lead to database index corruption if left unchecked.
+
+Affected Services
+- pgsql container
+- codeintel-db container
+- codeinsights-db container
+
+Only self-hosted customers using the Sourcegraph provided PostgreSQL container images are affected.
+
+Self-hosted customers using external databases, such as AWS RDS, GCP CloudSQL, or another self-managed solution are NOT affected.
+
+See our [PostgreSQL Collation Version Mismatch Resolution](/admin/postgresql_collation_version_mismatch_resolution) notes for more details.
+
 ## v6.0.0
 
 - Sourcegraph 6.0.0 no longer supports PostgreSQL 12, admins must upgrade to PostgreSQL 16. See our [postgres 12 end of life](/admin/postgres12_end_of_life_notice) notice! As well as [supporting documentation](/admin/postgres) and advisements on how to upgrade.
diff --git a/docs/admin/updates/pure_docker.mdx b/docs/admin/updates/pure_docker.mdx
@@ -9,6 +9,31 @@ Each section comprehensively describes the changes needed in Docker images, envi
 
 ## Unreleased
 
+## v6.2.2553
+
+### Known issues
+
+Customers running Sourcegraph versions prior to v6.2.2553 and using the Sourcegraph provided PostgreSQL containers may encounter PostgreSQL collation version mismatch warnings after upgrading to more recent Sourcegraph versions due to an underlying glibc version update.
+
+When logging into the database via psql or similar tools you may see the following warning:
+
+```shell
+WARNING: database "sg" has a collation version mismatch DETAIL: The database was created using collation version 2.40, but the operating system provides version 2.41.
+```
+
+Mismatched collation versions can lead to database index corruption if left unchecked.
+
+Affected Services
+- pgsql container
+- codeintel-db container
+- codeinsights-db container
+
+Only self-hosted customers using the Sourcegraph provided PostgreSQL container images are affected.
+
+Self-hosted customers using external databases, such as AWS RDS, GCP CloudSQL, or another self-managed solution are NOT affected.
+
+See our [PostgreSQL Collation Version Mismatch Resolution](/admin/postgresql_collation_version_mismatch_resolution) notes for more details.
+
 ## v5.2.6 ➔ v5.2.7
 
 As a template, perform the same actions as the following diff in your own deployment: [`Upgrade to v5.2.7`](https://github.com/sourcegraph/deploy-sourcegraph-docker/compare/v5.2.6...v5.2.7)
diff --git a/docs/technical-changelog.mdx b/docs/technical-changelog.mdx
@@ -1010,6 +1010,9 @@ Revert "[Backport 6.2.x] fix(agents): filter out empty diagnostic paths" `(PR #4
 
 - [kustomize](https://github.com/sourcegraph/deploy-sourcegraph-k8s/releases/tag/v6.2.2553)
 
+> Attention - This patch contains a known issue for self-hosted customers upgrading from previous versions of Sourcegraph whom are using the Sourcegraph provided PostgreSQL container images.
+> Please see our [PostgreSQL Collation Version Mismatch Resolution](/admin/postgresql_collation_version_mismatch_resolution) notes for more details.
+
 ### Features
 
 #### Source
