Failures on `teardown` are ignored and the test is reported as `passed`

[gqmelo]

When looking at a build where tests failed, I was surprised to see that in the Test analytics, a test was reported as passed even though it failed.

The reason is that the test failed on a fixture teardown and the plugin only considers failures on setup and call phases:

test-collector-python/src/buildkite_test_collector/pytest_plugin/buildkite_plugin.py

Line 50 in 8c1b40c

if report.when == 'call' or (report.when == 'setup' and report.failed):

Investigating the commit history, I couldn't find why this was done.

I can imagine an argument saying that if the test is checking a feature/desired behaviour, then even if the teardown fails, the feature/desired behaviour is correct and the test can be considered as passed.

But that doesn't really work well if we consider the purpose of Test analytics. A test whose teardown fails:

1. makes the whole build fail
2. can skip necessary cleanup could affect the next tests
3. could reveal flakiness in the setup/teardown logic
4. could be used as a way to ensure some post-conditions after the test runs (e.g. an autouse fixture ensuring that no test created a resource it wasn't supposed to).

So if the teardown fails, I would like to see which tests are decreasing my test suite reliability. Ignoring a teardown failure doesn't seem to have any benefit.

Is there a strong reason for the current behaviour?

[gchan]

@gqmelo thanks for bringing this issue to our attention and your detailed report.

#85 is in progress and should resolve the issue.