-
Notifications
You must be signed in to change notification settings - Fork 30
Description
Observed behavior
Following the current fix for #1394, the virtual USS (uss_qualifier) still does not reliably clean up its operational intents in scenarios.astm.utm.DownUSS (and likely the other scenario based on that scenario). In s323 below (using v0.27.0), one part of the cleanup fails and that stops the cleanup effort entirely. Cleanup should almost never stop when something can't be cleaned up because that prevents any additional cleanup from being performed. And indeed, in this case, the virtual USS op intent is never cleaned up because cleanup stopped before that portion of the cleanup could be attempted -- the virtual USS's operational intent then causes s324 to fail.
Test check
See s323 and s324
Difference from expected behavior
First, s323's cleanup should not have stopped -- at the failed check, it should have continued to the rest of the cleanup. I suspect this may be due to reusing a test step fragment for cleanup when we're actually doing something a little different and therefore may have different severities for the checks in the fragment. If the fragment is only used in cleanup, the checks should be fixed to only be Medium severity unless something really bad happens. If the fragment is used in both cleanup and the body of test scenarios, we should make a new fragment customized for cleanup, or else just put the relevant fragment content directly in the cleanup documentation.
Second, s324 should not have failed despite the problem in s323 because there is a setup step specifically to handle this situation. s324e10 claims operational intent cleanup was successful, but there are no queries that seem to relate to this check. Was the DSS queried for existing op intents and that check determined the area was clear because no op intents were found? If so, the query should be logged (and we could inspect it to see why the pre-existing op intent wasn't found). If not, what information is the successful check based on? Why was the op intent that caused scenario failure in s324e13 not found and removed in the "Clear operational intents created by virtual USS" test step?
Additional context
Note that in the nominal case, searching the DSS for virtual USS op intents will yield no op intents at all. It seems like that DSS query could be reused for the "Verify area is clear test step".