ENH: Reduce CTest verbosity to show output only for failing tests

[hjmjohnson]

Summary

Replace -VV (extra verbose) with -V (verbose) in all CI pipeline ctest -S invocations across 8 YAML files.

Background

Every ITK CI pipeline currently runs CTest in extra-verbose mode (-VV), which prints the full stdout/stderr of every test — regardless of whether it passed or failed. With ITK's 3000+ test suite, this generates enormous CI log output, making it difficult to locate actual failures when a build goes red.

Every pipeline also sets the environment variable CTEST_OUTPUT_ON_FAILURE=1, which is designed to print test output only for failing tests. However, this variable has been effectively dead code — the -VV flag unconditionally prints all output, overriding the output-on-failure behavior.

How CTest verbosity works

Flag	Behavior
(none)	Minimal output — just summary counts
-V	Verbose — prints test names, pass/fail status, and timing
-VV	Extra verbose — prints full stdout/stderr of every test
CTEST_OUTPUT_ON_FAILURE=1	Prints full output only for tests that fail (works with -V or no flag, overridden by -VV)

The CTEST_OUTPUT_ON_FAILURE environment variable works in both direct ctest invocations and ctest -S scripted/dashboard mode. The ctest_test() CMake command used internally by itk_common.cmake does not have an OUTPUT_ON_FAILURE keyword argument, but the environment variable is read by the CTest process regardless of invocation mode.

What this changes

• Before: -VV prints all 3000+ tests' output → large logs, failures buried in noise
• After: -V + CTEST_OUTPUT_ON_FAILURE=1 prints progress for all tests, full output only for failures → smaller logs, failures immediately visible

Files changed

File	Occurrences
.github/workflows/arm.yml	1
AzurePipelinesLinux.yml	3 (one per build config)
AzurePipelinesLinuxPython.yml	1
AzurePipelinesMacOS.yml	1
AzurePipelinesMacOSPython.yml	1
AzurePipelinesWindows.yml	1
AzurePipelinesWindowsPython.yml	1
AzurePipelinesBatch.yml	1

Test plan

• All pipelines already set CTEST_OUTPUT_ON_FAILURE: 1 in their env blocks
• CI builds pass on all platforms (Linux, macOS, Windows, ARM)
• Failing tests still produce full diagnostic output in logs

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR replaces -VV (extra-verbose) with -V (verbose) in all eight CI pipeline ctest -S invocations, allowing the already-configured CTEST_OUTPUT_ON_FAILURE=1 environment variable to take effect. The result is that CI logs will show test names and pass/fail status for every test but only print full stdout/stderr for failing tests — significantly reducing log noise on ITK's 3 000+ test suite while keeping failures immediately visible.

The change is mechanical and complete: all eight YAML files that call ctest -S have been updated, no other CI files run ctest, and every changed step already carries CTEST_OUTPUT_ON_FAILURE: 1 in its env: block.

Notable considerations:

• Output from passing tests will no longer appear in historical CI logs, which can make it harder to retrospectively debug intermittent failures that previously passed. Teams should be aware of this trade-off.
• The CTEST_OUTPUT_ON_FAILURE environment variable is confirmed to work in scripted/dashboard mode (ctest -S); it is read by ctest_test() internally regardless of invocation mode.
• No issues were found with the implementation — all pipelines (Linux, macOS, Windows, ARM) are handled consistently.

Confidence Score: 5/5

• This PR is safe to merge — it is a mechanical, low-risk verbosity reduction that makes a previously inert env var effective.
• The change is a single-character flag swap (-VV → -V) repeated identically across 8 files. Every affected step already carries CTEST_OUTPUT_ON_FAILURE: 1, so the intended semantics (full output only on failure) are already in place and will simply activate. No test logic, build configuration, or application code is modified. All CI platforms (Linux, macOS, Windows, ARM) are covered and none were missed.
• No files require special attention.

Important Files Changed

Filename	Overview
.github/workflows/arm.yml	Single -VV → -V replacement at line 177; CTEST_OUTPUT_ON_FAILURE: 1 already present in the same step's env: block.
Testing/ContinuousIntegration/AzurePipelinesLinux.yml	Three -VV → -V replacements (lines 114, 205, 294) covering the Linux, LinuxLegacyRemoved, and LinuxCxx20 jobs; each step already sets CTEST_OUTPUT_ON_FAILURE: 1.
Testing/ContinuousIntegration/AzurePipelinesLinuxPython.yml	Single -VV → -V replacement; CTEST_OUTPUT_ON_FAILURE: 1 already set. The lengthy -E exclusion filter is preserved correctly.
Testing/ContinuousIntegration/AzurePipelinesMacOS.yml	Single -VV → -V replacement; CTEST_OUTPUT_ON_FAILURE: 1 already set.
Testing/ContinuousIntegration/AzurePipelinesMacOSPython.yml	Single -VV → -V replacement; CTEST_OUTPUT_ON_FAILURE: 1 already set.
Testing/ContinuousIntegration/AzurePipelinesWindows.yml	Single -VV → -V replacement on a Windows script: step; Azure Pipelines env: block sets CTEST_OUTPUT_ON_FAILURE: 1 for cmd-based steps as well.
Testing/ContinuousIntegration/AzurePipelinesWindowsPython.yml	Single -VV → -V replacement; CTEST_OUTPUT_ON_FAILURE: 1 already set, -L Python label filter preserved.
Testing/ContinuousIntegration/AzurePipelinesBatch.yml	Single -VV → -V replacement in the Windows batch pipeline; CTEST_OUTPUT_ON_FAILURE: 1 already set.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["CI pipeline triggers\n(push / PR)"] --> B["ctest -S dashboard.cmake -V -j N"]
    B --> C{Test result?}
    C -- Pass --> D["Print: test name + status + timing\n(no stdout/stderr captured)"]
    C -- Fail --> E["CTEST_OUTPUT_ON_FAILURE=1\nPrint: test name + status + full stdout/stderr"]
    D --> F["CTest summary"]
    E --> F
    F --> G["ci_addons: format JUnit XML\n(Azure Pipelines only)"]
    G --> H["PublishTestResults\n(Azure Pipelines only)"]

    style E fill:#f96,color:#000
    style D fill:#9c9,color:#000

Loading

_{Last reviewed commit: "ENH: Reduce CTest ve..."}

[thewtex]

👍 thanks, Hans!