Migration from vstest

[ladeak]

I would like to migrate a solution with 100 test projects from vstest to MTP. The solution contains test projects either xUnit or MSTest and some integration tests.

When I enable MTP in the global.json (and migrate the test command), I receive errors on some of the switches not supported by MSTest or some of the switches not supported by xUnit. For example, the "--filter" vs --filter-trait".

Or similar issues with reporting in junit format: one says --report-spekt-junit the other has options also for --report-junit

(This is a continuation of a previous issue of #6484 )

What are the suggested approaches for migrating?

[Evangelink]

There is currently no ideal option for partial migration. May I ask what's the challenge for you to do the upgrade of all projects at once? From our collaboration/partnership with some big teams, it's been usually pretty straightforward work (replacing some NuGet packages, adding some properties in Directory.Build.props and updating the dotnet test CLI arguments).

[Youssef1313]

There is currently no ideal option for partial migration

I don't think the ask here is about mixing VSTest and MTP in the same solution (i.e, partial migration).

The ask here is using both xUnit and MSTest in the same solution.

For filtering, this will improve with xunit/xunit#3466, so that you can use --filter when running on a solution and it will be supported by both MSTest and xUnit, similar to how that was possible with VSTest.

For reporting, --report-junit is xunit-specific. @bradwilson were you renaming these in xunit.v3 4.x?

My advise there would be to add spekt logger package to both the MSTest and the xUnit projects, and always use --report-spekt-junit which should just work. If Spekt logging doesn't work with xUnit for any reason, I would suggest opening an issue in the corresponding product repo in https://github.com/spekt/testlogger

[ladeak]

Yes it is correct, I would like to do a full migration, but it includes both xUnit and MSTest projects (and some integration tests in both of those projects that I'd like to filter).

Thank you for the details, so for junit going with --report-spekt-junit, for the --filter there is a story in place - which probably I would need to wait on the referenced ticket.

I had a weird behavior in terms that I am trying to set <EnableMSTestRunner> in a Directory.Build.targets file, but somehow this approach is not recognized by MTP showing an error that the tests do not enable the test runner. However, when I add <EnableMSTestRunner> to the actual projects's <PropertyGroup> it works.
So eventually (because I need to touch all MSTest projects) I ended up updating the sdk of the projects itself: <Project Sdk="MSTest.SDK/4.2.1"> - I found this weird because adding the other nuget packages (for code coverage and hangdump) seems to work fine through the Directory.Build.targets file.

On MSTest side, I am using a Microsoft.VisualStudio.TestTools.UnitTesting.DoNotParallelizeAttribute - is this still the right approach? (asking because the namespace looks suspicious and I have not found other alternative, while I have some suspicious failures probably due to parallel execution in some projects).

[Youssef1313]

Directory.Build.targets might be late to set EnableMSTestRunner. Have you tried Directory.Build.props instead?

PackageReferences are generally fine to have in Directory.Build.targets though.

For MSTest, parallelization is already disabled by default. The namespace isn't suspicious though, that's where almost all MSTest APIs live.

[ladeak]

Directory.Build.props did not work to my memory either, it is in each project's csproj a flag defining to use xunit or nunit. But <Project Sdk="MSTest.SDK/4.2.1"> was fine for me. Still experimenting if the MSTest.SDK/4.2.1 part can be moved the Diretory.Build.props.

I managed to resolve the issues attributed for parallelization - it had nothing to do with it.

It seems at the moment that --filter and --filter-not-trait is the main difference for dotnet test command. I ended up with 2 temporary slnx files excluding one or the other test projects. Not ideal, but seems to be a temporary solution until the referenced filters are addressed.

I am going to verify tomorrow code coverage result files and the junit file.

[bradwilson]

For reporting, --report-junit is xunit-specific. @bradwilson were you renaming these in xunit.v3 4.x?

Yes. All reports in 4.0 have been renamed to prevent naming collisions:

  --report-xunit-ctrf              Enable generating CTRF (JSON) report
  --report-xunit-ctrf-filename     The name of the generated CTRF (JSON) report
  --report-xunit-html              Enable generating HTML report
  --report-xunit-html-filename     The name of the generated HTML report
  --report-xunit-junit             Enable generating JUnit (XML) report
  --report-xunit-junit-filename    The name of the generated JUnit (XML) report
  --report-xunit-nunit             Enable generating NUnit v3 (XML) report
  --report-xunit-nunit-filename    The name of the generated NUnit v3 (XML) report
  --report-xunit-trx               Enable generating TRX (XML) report
  --report-xunit-trx-filename      The name of the generated TRX (XML) report
  --report-xunit-xml               Enable generating xUnit.net v2+ (XML) report
  --report-xunit-xml-filename      The name of the generated xUnit.net v2+ (XML) report

This is already available in the prerelease builds on NuGet.

The plan is also for --filter to be part of 4.0, though it has not yet been merged.

[ladeak]

In a more general approach to would it make sense to have the ablilty to pass arguments to dotnet test command in a way that it is only forwarded to certain frameworks?
For example, in a solution with multiple projects, when I run tests I would like to pass in --report-xunit-html for xUnit but I would not want projects that is use MSTest to fail due to an unknown argument.

[bradwilson]

I think the best solution is to come up with a third party HTML reporting tool. Our HTML is so terrible you can't possibly love it 😂 but it wouldn't be that hard for someone to port it from the xUnit.net object model to the MTP object model and publish it as a NuGet package. There was some talk about whether that should be done with our JUnit reporting as well.

[bradwilson]

FWIW, we would probably still continue to ship our own versions of these even if they got generalized. We do that today for TRX, because the MTP object model is not as complete as the xUnit.net object model (in particular, no way to report "not run" tests in a standardized way, which was a rejected feature request). So our TRX reports include "not run" tests in the TRX output, whereas the Microsoft provided TRX reporter does not (I think probably reports them as "skipped").

[Youssef1313]

Directory.Build.props did not work to my memory either

If EnableMSTestRunner isn't working from Directory.Build.props, I think this is not expected. It would help to have some repro for investigation.

In a more general approach to would it make sense to have the ablilty to pass arguments to dotnet test command in a way that it is only forwarded to certain frameworks?

You can try to experiment with something like: dotnet test -p:XUnitSpecificArgs=".." -p:MSTestSpecificArgs="..", noting that XUnitSpecificArgs and MSTestSpecificArgs are just arbitrary names and are not well-known by any tooling, so you will then need to manually figure out per your scenario how to apply these to specific frameworks and append them to TestingPlatformCommandLineArguments. If you went with MSTest.Sdk already, I think you can have something like the following in Directory.Build.targets:

<PropertyGroup>
    <TestingPlatformCommandLineArguments Condition="'$(UsingMSTestSdk)' == 'true'">$(TestingPlatformCommandLineArguments) $(MSTestSpecificArgs)</TestingPlatformCommandLineArguments>
    <TestingPlatformCommandLineArguments Condition="'$(UsingMSTestSdk)' != 'true'">$(TestingPlatformCommandLineArguments) $(XUnitSpecificArgs)</TestingPlatformCommandLineArguments>
</PropertyGroup>

MSTest.TestAdapter might also consider having <IsUsingMSTest>true</IsUsingMSTest> in its NuGet props to allow similar scenarios even when not using MSTest.Sdk. xUnit might also consider having IsUsingXUnit in its package props.

[bradwilson]

I can think of a few values I could set:

NuGet package	$(XunitTestProject)	$(XunitTestProjectAOT)	$(XunitTestProjectMTP)
xunit.v3.core	true		true
xunit.v3.core.mtp-v2	true		true
xunit.v3.core.mtp-off	true
xunit.v3.core.aot	true	true	true
xunit.v3.core.aot.mtp-v2	true	true	true
xunit.v3.core.aot.mtp-off	true	true

These would be specifically centered on the xunit.v3.core packages, since inclusion of that package is what we consider to be the indicator that you are writing a test project (for example, it's where we enforce requirements like <OutputType>Exe</OutputType> and <UseAppHost>true</UseAppHost> as well as ensuring that the project language is C# for AOT).

[Youssef1313]

I wouldn't introduce a specific property for MTP. It could easily interfere with when users disable it via IsTestingPlatformApplication. Also, IsTestingPlatformApplication today can be set any time during evaluation, which can be very late.

[ladeak]

The idea on <TestingPlatformCommandLineArguments> looks very promising.

[ladeak]

$(TestingPlatformCommandLineArguments) $(MSTestSpecificArgs) $(TestingPlatformCommandLineArguments) $(XUnitSpecificArgs) MSTest.TestAdapter might also consider having `true` in its NuGet props to allow similar scenarios even when not using MSTest.Sdk. xUnit might also consider having `IsUsingXUnit` in its package props.

I managed to get everything working with this setup, and even managed a single dotnet test command that includes xUnit and MSTests. The extra bit from xUnit side was adding an ignore flag for error code 8 (not test in the assembly, because they were all integration tests).

Loggers, filtering seem to work fine too. I am closing some gaps with code coverage, as I see coverlet.MTP is also available by now for coverage.

Thank you @Youssef1313 and @bradwilson for the support.