Unable to build with precommit profile

[pavelhoral]

Unit tests in opendj-server-legacy module are only executed with precommit maven profile. We don't have this profile in our CI pipeline which is probably mistake.

We need to fix unit tests so that they are passing and add this profile to CI pipeline.

[pavelhoral]

Current output of running (macOS)

mvn -Pmetrics,precommit -Dcheckstyle.skip verify -pl opendj-server-legacy

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M7:verify (default) on project opendj-server-legacy: 
[ERROR] 
[ERROR] Please refer to /Users/horal/Documents/repository/wrends/opendj-server-legacy/target/failsafe-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] org/opends/server/backends/jeb/JEBackend
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:701)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:311)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:268)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1334)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1167)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:931)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:301)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:211)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:165)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:157)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:121)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:127)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
[ERROR] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
[ERROR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

[pavelhoral]

I was able to make it run with a few tweaks. I will create PR soon-ish, but first I need to investigate test failures:

Finished at: Sun Feb 05 12:30:21 CET 2023
# Test classes: 287
# Test classes interleaved: 0
# Test methods: 3212
# Tests passed: 24213
# Tests failed: 85

Also it is really slow and I am not sure if there is a reason for it:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23:55 min
[INFO] Finished at: 2023-02-05T12:30:33+01:00
[INFO] ------------------------------------------------------------------------

And finally it seems that some tests are not made for Windows as there were failures related to backslashes in filesystem paths.

[pavelhoral]

Running on Linux gives different results:

# Test classes: 287
# Test classes interleaved: 0
# Test methods: 3230
# Tests passed: 24273
# Tests failed: 52

And for some reason ends with SUCCESS and in much faster time:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:38 min
[INFO] Finished at: 2023-02-06T10:47:25+01:00
[INFO] ------------------------------------------------------------------------

[pavelhoral]

Seems like a lot of tests are failing due to this sneaky TestNG addition. Kind of wants me to throw TestNG away 😠

Failure example (org.opends.server.types.Attribute implements Iterable):

java.lang.AssertionError: Iterators differ at element [0]: description:0000000000000001000000000000:repl:value1 != description:0000000000000002000000000000:del:value3 expected [description:0000000000000001000000000000:repl:value1] but found [description:0000000000000002000000000000:del:value3]

[pavelhoral]

Example invocation that ends with error even though individual tests are successful:

mvn failsafe:integration-test -pl opendj-server-legacy -P precommit -Dit.test=SchemaReplicationTest,StateMachineTest

I am not sure how to enable debug logging in tests to actually see what is happening.

[pavelhoral]

I have invested two more days into this issue. There is no single cause for the broken tests so there is no silver bullet fix that will make precommit profile work.

The correct way would be to slowly fix one test after another. Doing something like this is pretty huge task - it might require non-trivial changes (plus working with TestNG alone is pretty developer unfriendly - at least for me). Also quite a lot (I mean a lot) of tests are simply broken and logging ton of errors, but they somehow end with success. So even detecting non working tests is a challenge on its own.

Of course fixing one test at a time won't necessarily lead to working precommit profile because those tests are leaking resources and there is a lot of shared state. Running one test class after another might lead to a new error that is not present when each class is run on its own.

There are approximately 350 test classes. We can add @Ignore annotations to all of them and then slowly remove
them one by one after refactoring those tests so that the test architecture is correct and tests have no side effects. That is the only way to enable precommit profile and continue with major release. Another alternative is to simply keep precommit disabled.