DSS continues processing after timeout

[gc-2412]

Overview:
ANRA observed an instance in which the DSS seemed to continue processing a request even after responding with a timeout code.

DSS Instance:

• DSS Version (FAA Keysite SDD 1.4)
  • DSS core service version v0.20.0
  • CockroachDB version v24.1.3

Events:

The DSS timed out after ~10s while processing a PUT /dss/v1/operational_intent_references/{entityid}/{ovn}
request for the operation: 4748e25c-d3fb-483f-90e2-59568f446e4d
The request was an attempt to update the state to “Activated”

Request Url: https://dss.us.interuss.opensky.dev/dss/v1/operational_intent_references/4748e25c-d3fb-483f-90e2-59568f446e4d/9FI3V7CtbSn5quHEPv5Bn8S8a40LWKYzrFFh8a5QE9A_
Response Exception Status: ProtocolError
Response Duration: 10141.0102 ms
Via: 1.1 google
Date: Thu, 12 Mar 2026 22:04:44 GMT
Exception: The remote server returned an error: (503) Service Unavailable.

upstream connect error or disconnect/reset before headers. reset reason: connection termination

The next activate request was made at Thu Mar 12 2026 22:04:52 GMT. At this point the DSS despite timing out, completed the request and updated the operation to “Activated” state and assigned a new OVN.

Request Url:
https://dss.us.interuss.opensky.dev/dss/v1/operational_intent_references/4748e25c-d3fb-483f-90e2-59568f446e4d/9FI3V7CtbSn5quHEPv5Bn8S8a40LWKYzrFFh8a5QE9A_
Response Exception Status: ProtocolError
Response Duration: 301.9925 ms
Date: Thu, 12 Mar 2026 22:04:52 GMT 
Via: 1.1 google 
Alt-Svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000 
Content-Type: application/json; charset=utf-8 
Content-Length: 193 
Exception: The remote server returned an error: (409) Conflict.

{"message":"Current version is 1larMLymMqUs97vaNL7ZcWhgEVHp1bkr9xCqD-oZHM8_ but client specified version 9FI3V7CtbSn5quHEPv5Bn8S8a40LWKYzrFFh8a5QE9A_ (E:447e9272-9b12-4695-990d-763fb26973ee)"}

The response of a POST /dss/v1/operational_intent_references/query proving that the state had already transitioned to Activated

Request Url: https://dss.us.interuss.opensky.dev/dss/v1/operational_intent_references/query
Response Code: 200 - OK
Response Duration: 92.4041 ms
Date: Thu, 12 Mar 2026 22:04:47 GMT 
Via: 1.1 google 
Alt-Svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000 
Transfer-Encoding: chunked 
Content-Type: application/json; charset=utf-8

-- omitted for brevity --
{"id":"4748e25c-d3fb-483f-90e2-59568f446e4d","manager":"ANRA","uss_availability":"Unknown","version":2,"state":"Activated","ovn":"1larMLymMqUs97vaNL7ZcWhgEVHp1bkr9xCqD-oZHM8_","time_start":{"value":"2026-03-12T22:08:52Z","format":"RFC3339"},"time_end":{"value":"2026-03-12T22:23:41Z","format":"RFC3339"},"uss_base_url":"https://keysite.flyanra.net/flight-auth","subscription_id":"c7eeae0f-4fc2-4e20-aadd-0eb44f01155f"}
-- omitted for brevity --

This observation indicates that despite timing out, the DSS continued processing the request.

[the-glu]

Hello,

You seem to use a proxy between you and the DSS (due to the 503 responses), so that complicates things.

If the proxy times out (and finishes the transaction between you and the proxy), that doesn't mean the transaction between the proxy and the DSS is aborted; it may continue in the background.

CLIENT              PROXY                DSS (Server)
      |                  |                    |
      |-- HTTP Request ->|                    |
      |                  |-- HTTP Request --->|
      |                  |                    |  <-- DSS starts 
      |                  |      Processing    |      heavy task...
      |      (WAIT)      |                    |
      |                  |                    |
      X <--- TIMEOUT ----|                    |  <-- Proxy cuts the
   (Connexion closed)    |                    |      client connection
      |                  |                    |
      |                  |                    |  <-- DSS remains
      |   [Proxy frees]  |                    |      UNAWARE and
      |   [resources]    |                    |      continues...
      |                  |                    |
      |                  |<--- FINISH TX -----|  <-- Response is sent
      |                  |   (Discarded)      |      but ignored

It's also possible that the DSS finished its SQL transaction and finished responding, but didn't have time to answer as the proxy just closed the connection as well.

If the proxy is set to the same (or lesser) timeout than the DSS uses, you may run into those race conditions.

Does the proxy have a smaller timeout than the DSS?

If it's not due to timing with the proxy, then it would be an issue in the DSS not canceling the transaction when it should.

I tried to simulate some timeouts manually and it does seem to work as expected.

However, when debugging, it seems that transactions in the SCD datastore don't have a default timeout and there is no explicit timeout. This means only those from the HTTP server will apply, and when I tested those, they don't cancel database transactions.

We should probably have an automatic timeout on every HTTP query based on the timeout flag, to ensure it's applied everywhere, I will propose a PR