Skip to content

Some test scenarios do not clean up reliably #1400

@BenjaminPelletier

Description

@BenjaminPelletier

Observed behavior
Following the current fix for #1394, the virtual USS (uss_qualifier) still does not reliably clean up its operational intents in scenarios.astm.utm.DownUSS (and likely the other scenario based on that scenario). In s323 below (using v0.27.0), one part of the cleanup fails and that stops the cleanup effort entirely. Cleanup should almost never stop when something can't be cleaned up because that prevents any additional cleanup from being performed. And indeed, in this case, the virtual USS op intent is never cleaned up because cleanup stopped before that portion of the cleanup could be attempted -- the virtual USS's operational intent then causes s324 to fail.

s323.html

s324.html

Test check
See s323 and s324

Difference from expected behavior
First, s323's cleanup should not have stopped -- at the failed check, it should have continued to the rest of the cleanup. I suspect this may be due to reusing a test step fragment for cleanup when we're actually doing something a little different and therefore may have different severities for the checks in the fragment. If the fragment is only used in cleanup, the checks should be fixed to only be Medium severity unless something really bad happens. If the fragment is used in both cleanup and the body of test scenarios, we should make a new fragment customized for cleanup, or else just put the relevant fragment content directly in the cleanup documentation.

Second, s324 should not have failed despite the problem in s323 because there is a setup step specifically to handle this situation. s324e10 claims operational intent cleanup was successful, but there are no queries that seem to relate to this check. Was the DSS queried for existing op intents and that check determined the area was clear because no op intents were found? If so, the query should be logged (and we could inspect it to see why the pre-existing op intent wasn't found). If not, what information is the successful check based on? Why was the op intent that caused scenario failure in s324e13 not found and removed in the "Clear operational intents created by virtual USS" test step?

Additional context
Note that in the nominal case, searching the DSS for virtual USS op intents will yield no op intents at all. It seems like that DSS query could be reused for the "Verify area is clear test step".

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions