pytest-subtests: OnlyMutedFailures() returns false due to unrecognized subtest statuses

[jasonwbarnett]

Description

When using pytest with pytest-subtests, bktec's muted test exit code suppression (OnlyMutedFailures()) does not work. The build exits with code 1 even when the only test failure is from a muted test.

Environment

• bktec version: v2.3.0
• Test runner: pytest with pytest-subtests and pytest-xdist
• buildkite-test-collector for pytest

Root Cause Analysis

I traced through the bktec source code and identified the issue:

1. RunResult.Status() returns RunStatusUnknown

In internal/runner/run_result.go:168-186, Status() computes the run status by counting tests in known status buckets:

func (r *RunResult) Status() RunStatus {
    if r.error != nil { return RunStatusError }
    if len(r.tests) == 0 { return RunStatusUnknown }
    if len(r.FailedTests()) > 0 { return RunStatusFailed }
    if r.passedTestsCount()+r.skippedTestsCount()+len(r.FailedMutedTests()) == len(r.tests) {
        return RunStatusPassed
    }
    return RunStatusUnknown  // ← falls through here
}

The only recognized TestStatus values are "passed", "failed", and "skipped" (internal/runner/test_result.go:7-11).

When pytest-subtests is used, the buildkite-test-collector pytest plugin writes subtest results to the --json output file. These subtest results may have statuses that don't match the three recognized values. Since these tests are recorded in r.tests but don't count toward passedTestsCount, skippedTestsCount, or FailedMutedTests, the sum check fails and Status() falls through to RunStatusUnknown.

2. OnlyMutedFailures() bails out on RunStatusUnknown

In internal/runner/run_result.go:145-148:

func (r *RunResult) OnlyMutedFailures() bool {
    if r.Status() == RunStatusError || r.Status() == RunStatusUnknown {
        return false  // ← always false when Status() is Unknown
    }
    // ...
}

3. Exit code suppression doesn't fire

In internal/command/run.go:102-110:

if exitError.ExitCode() == 1 && runResult.OnlyMutedFailures() {
    return nil  // ← never reached because OnlyMutedFailures() returned false
}
return fmt.Errorf("%s exited with error: %w", testRunner.Name(), runErr)  // ← exit code 1 propagated

Observed Behavior

In a real CI build:

• pytest reported: 2 failed, 307 passed, 48 skipped, 331 warnings, 735 subtests passed
• The "2 failed" are both from the same muted test (1 SUBFAILED subtest + 1 FAILED parent from pytest-subtests)
• No bktec report was printed (confirming Status() returned RunStatusUnknown and printReport returned early)
• bktec output: pytest exited with error: exit status 1
• Build failed despite the only failure being a muted test

Expected Behavior

When the only test failures are from muted tests, bktec should exit with code 0 regardless of whether pytest-subtests is used.

Suggested Fix

The Status() function could handle unrecognized test statuses more gracefully. Options:

1. Ignore unrecognized statuses in the sum check — only count tests with recognized statuses toward the total
2. Treat unrecognized statuses as a fourth bucket — add them to the sum so the accounting still works
3. Map subtest statuses to parent statuses — normalize any unrecognized status to the closest standard one

Option 1 seems simplest and least risky — the sum check at the end of Status() would exclude tests with unrecognized statuses from both sides of the equation.

Reproduction

1. Create a pytest test suite using pytest-subtests
2. Mute a test that has subtests in Buildkite Test Engine
3. Run with bktec v2.3.0 and --json={{resultPath}}
4. Observe that when the muted test fails, bktec exits with code 1 instead of 0

[nprizal]

These subtest results may have statuses that don't match the three recognized values.

Hi @jasonwbarnett do you know how I can reproduce the subtests with "unknown" results?
I tried to reproduce using the steps as you described, but it correctly handles the muting because all tests has known results.

Also, does it happen for subtests only or for normal tests as well?

[nprizal]

Right, I think I’ve identified the problem. Apparently, there’s a bug in the test collector. When a test skipped during the setup phase the result isn’t marked as “skipped” (it’s nil). Therefore, when bktec parses the result the test results become Unknown.

How to reproduce:

1. Create dummy pytest file

import pytest

# mute this test in Test Engine
def test_with_subtests(subtests):
    with subtests.test(msg="subtest 1 failed"):
        assert 1 == 2
    with subtests.test(msg="subtest 2 passed"):
        assert 1 == 1

    with subtests.test(msg="subtest 2"):
        pytest.skip("skipping subtest 2")

@pytest.mark.skip
def test_my_test():
    assert 1 == 1

2. Run pytest against the dummy file using bktec
3. bktec exits with 1, and no report is printed

[nprizal]

I’ve opened a PR to fix a bug in the test collector. This fix will mark skipped tests as “skipped”, which will be included in the calculation for RunStatus.

[nprizal]

This is not a bug in the Test Engine Client, so I will close this issue.