BUG: Server crashing on DB connection drop

[vukilelanga]

Lifecycle's server has been dropping the connection to the database, which causes the server to crash. Client has raised the issue of the app being unavailable.

[vukilelanga]

Update:

• Kotahi Test deployment's database has been moved back to Digitalocean for testing.
• Another option: update coko server to v5, which includes an updated pg-boss (according to this support ticket)
• Also: update coko server/kotahi server to handle db connection errors

[vukilelanga]

Incident on bats-server:

2026-04-28 16:00:38.803 | ⛏   Coko server init tasks  ⛏ |  
-- | -- | --
  |   | 2026-04-28 16:00:29.685 | 2026/04/28 14:00:29 INFO SSH listening listen_address=[fdaa:4c:20a:a7b:2a9:205f:635:2]:22 |  
  |   | 2026-04-28 16:00:29.420 | Machine started in 1.385s |  
  |   | 2026-04-28 16:00:29.297 | INFO [fly api proxy] listening at /.fly/api |  
  |   | 2026-04-28 16:00:29.284 | INFO Preparing to run: `docker-entrypoint.sh yarn coko-server start` as node |  
  |   | 2026-04-28 16:00:29.185 | INFO Starting init (commit: 350f2667)... |  
  |   | 2026-04-28 16:00:28.213 | 2026-04-28T14:00:28.213106401 [01KMD6981638F5PK47FTT5GXGN:main] Listening on API socket ("/fc.sock"). |  
  |   | 2026-04-28 16:00:28.212 | 2026-04-28T14:00:28.212706824 [01KMD6981638F5PK47FTT5GXGN:main] Running Firecracker v1.14.2 |  
  |   | 2026-04-28 16:00:27.258 | [86400.215356] reboot: Restarting system |  
  |   | 2026-04-28 16:00:27.253 | INFO Starting clean up. |  
  |   | 2026-04-28 16:00:27.236 | INFO Main child exited normally with code: 1 |  
  |   | 2026-04-28 16:00:26.455 | 16: 0x1959ef6  [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.448 | 15: 0x1529950 v8::internal::Runtime_AddDictionaryProperty(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.447 | 14: 0x13dde0a v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.444 | 13: 0x13d9059 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.443 | 12: 0x13d8453 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::AllocationType) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.435 | 11: 0x13d822d v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New<v8::internal::Isolate>(v8::internal::Isolate*, int, v8::internal::AllocationType, v8::internal::MinimumCapacity) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.428 | 10: 0x10b5404 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.427 | 9: 0x10b529c v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.410 | 8: 0x10c3b26 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.410 | 7: 0x10e7405 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.410 | 6: 0x10e6271 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.410 | 5: 0x1110158 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.410 | 4: 0x10f82d5  [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.410 | 3: 0xee65d7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.410 | 2: 0xee62f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.404 | 1: 0xb76db1 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/local/bin/node] |  
  |   | 2026-04-28 16:00:26.372 | ----- Native stack trace ----- |  
  |   | 2026-04-28 16:00:26.372 | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory |  
  |   | 2026-04-28 16:00:26.372 | <--- JS stacktrace ---> |  
  |   | 2026-04-28 16:00:26.372 | [669:0x2da4ac10] 86397272 ms: Mark-Compact (reduce) 479.6 (490.1) -> 479.1 (490.9) MB, 429.91 / 0.00 ms  (average mu = 0.912, current mu = 0.005) allocation failure; scavenge might not succeed |  
  |   | 2026-04-28 16:00:26.372 | [669:0x2da4ac10] 86396840 ms: Mark-Compact (reduce) 479.0 (490.1) -> 478.6 (490.1) MB, 331.22 / 0.00 ms  (average mu = 0.957, current mu = 0.024) allocation failure; scavenge might not succeed |  
  |   | 2026-04-28 16:00:26.372 | <--- Last few GCs ---> |  
  |   | 2026-04-28 16:00:01.299 | } |  
  |   | 2026-04-28 16:00:01.299 | client: 'postgres' |  
  |   | 2026-04-28 16:00:01.299 | }, |  
  |   | 2026-04-28 16:00:01.299 | '    at TCP.onStreamRead (node:internal/stream_base_commons:191:23)' |  
  |   | 2026-04-28 16:00:01.299 | '    at Readable.push (node:internal/streams/readable:392:5)\n' + |  
  |   | 2026-04-28 16:00:01.299 | '    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n' + |  
  |   | 2026-04-28 16:00:01.299 | '    at addChunk (node:internal/streams/readable:561:12)\n' + |  
  |   | 2026-04-28 16:00:01.299 | '    at Socket.emit (node:events:524:28)\n' + |  
  |   | 2026-04-28 16:00:01.299 | '    at Socket.<anonymous> (/home/node/app/node_modules/pg-protocol/dist/index.js:11:42)\n' + |  
  |   | 2026-04-28 16:00:01.299 | '    at Parser.parse (/home/node/app/node_modules/pg-protocol/dist/parser.js:35:38)\n' + |  
  |   | 2026-04-28 16:00:01.299 | '    at Parser.handlePacket (/home/node/app/node_modules/pg-protocol/dist/parser.js:122:29)\n' + |  
  |   | 2026-04-28 16:00:01.299 | '    at Parser.parseErrorMessage (/home/node/app/node_modules/pg-protocol/dist/parser.js:283:98)\n' + |  
  |   | 2026-04-28 16:00:01.299 | originalStack: `error: update "manuscripts" set "type" = $1, "is_hidden" = $2, "updated" = $3 where "created" < $4 and "status" = $5 and "group_id" = $6 and not "is_hidden" = $7 and (submission->>'$customStatus' = '' or submission->>'$customStatus' IS NULL) - invalid input syntax for type timestamp with time zone: "0NaN-NaN-NaNTNaN:NaN:NaN.NaN+NaN:NaN"\n` + |  
  |   | 2026-04-28 16:00:01.299 | routine: 'DateTimeParseError', |  
  |   | 2026-04-28 16:00:01.299 | line: '4140', |  
  |   | 2026-04-28 16:00:01.299 | file: 'datetime.c', |  
  |   | 2026-04-28 16:00:01.299 | constraint: undefined, |  
  |   | 2026-04-28 16:00:01.299 | dataType: undefined, |  
  |   | 2026-04-28 16:00:01.299 | column: undefined, |  
  |   | 2026-04-28 16:00:01.299 | table: undefined, |  
  |   | 2026-04-28 16:00:01.299 | schema: undefined, |  
  |   | 2026-04-28 16:00:01.299 | where: "unnamed portal parameter $4 = '...'", |  
  |   | 2026-04-28 16:00:01.299 | internalQuery: undefined, |  
  |   | 2026-04-28 16:00:01.299 | internalPosition: undefined, |  
  |   | 2026-04-28 16:00:01.299 | position: undefined, |  
  |   | 2026-04-28 16:00:01.299 | hint: undefined, |  
  |   | 2026-04-28 16:00:01.299 | detail: undefined, |  
  |   | 2026-04-28 16:00:01.299 | code: '22007', |  
  |   | 2026-04-28 16:00:01.299 | severity: 'ERROR', |  
  |   | 2026-04-28 16:00:01.299 | length: 197, |  
  |   | 2026-04-28 16:00:01.299 | at async Job.fn [as job] (/home/node/app/utils/jobUtils.js:48:11) { |  
  |   | 2026-04-28 16:00:01.299 | at async archiveOldManuscripts (/home/node/app/controllers/manuscript/manuscriptCommsUtils.js:69:25) |  
  |   | 2026-04-28 16:00:01.299 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) |  
  |   | 2026-04-28 16:00:01.299 | at QueryBuilder.execute (/home/node/app/node_modules/objection/lib/queryBuilder/QueryBuilder.js:446:28) |  
  |   | 2026-04-28 16:00:01.299 | at doExecute (/home/node/app/node_modules/objection/lib/queryBuilder/QueryBuilder.js:1099:31) |  
  |   | 2026-04-28 16:00:01.299 | at buildKnexQuery (/home/node/app/node_modules/objection/lib/queryBuilder/QueryBuilder.js:1240:63) |  
  |   | 2026-04-28 16:00:01.299 | at Function.value (/home/node/app/node_modules/knex/lib/knex-builder/make-knex.js:92:29) |  
  |   | 2026-04-28 16:00:01.299 | at Object.queryBuilder (/home/node/app/node_modules/knex/lib/knex-builder/make-knex.js:112:26) |  
  |   | 2026-04-28 16:00:01.299 | nativeError: error: update "manuscripts" set "type" = $1, "is_hidden" = $2, "updated" = $3 where "created" < $4 and "status" = $5 and "group_id" = $6 and not "is_hidden" = $7 and (submission->>'$customStatus' = '' or submission->>'$customStatus' IS NULL) - invalid input syntax for type timestamp with time zone: "0NaN-NaN-NaNTNaN:NaN:NaN.NaN+NaN:NaN" |  
  |   | 2026-04-28 16:00:01.299 | at async Job.fn [as job] (/home/node/app/utils/jobUtils.js:48:11) { |  
  |   | 2026-04-28 16:00:01.299 | at async archiveOldManuscripts (/home/node/app/controllers/manuscript/manuscriptCommsUtils.js:69:25) |  
  |   | 2026-04-28 16:00:01.299 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) |  
  |   | 2026-04-28 16:00:01.299 | at QueryBuilder.execute (/home/node/app/node_modules/objection/lib/queryBuilder/QueryBuilder.js:449:20) |  
  |   | 2026-04-28 16:00:01.299 | at handleExecuteError (/home/node/app/node_modules/objection/lib/queryBuilder/QueryBuilder.js:1140:32) |  
  |   | 2026-04-28 16:00:01.299 | at wrapError (/home/node/app/node_modules/db-errors/lib/dbErrors.js:19:14) |  
  |   | 2026-04-28 16:00:01.299 | DataError: update "manuscripts" set "type" = $1, "is_hidden" = $2, "updated" = $3 where "created" < $4 and "status" = $5 and "group_id" = $6 and not "is_hidden" = $7 and (submission->>'$customStatus' = '' or submission->>'$customStatus' IS NULL) - invalid input syntax for type timestamp with time zone: "0NaN-NaN-NaNTNaN:NaN:NaN.NaN+NaN:NaN" |  
  |   | 2026-04-28 16:00:00.881 | Importing manuscripts. Triggered by system |  
  |   | 2026-04-28 16:00:00.880 | Running scheduler for importing and archiving Manuscripts at 2026-04-28T14:00:00.880Z

[vukilelanga]

Waiting on #407