Potential deadlock in TransactionServiceImp when a commit is retried?

[momsen]

Describe the bug
When a custom XA resource does fail in a commit (one phase) once and signals that the commit may be retried, the TransactionServiceImp hangs in the shutdown procedure. The resource does successful commit the second time.
The issue probably arises, because TransactionServiceImp#removeCoordinator and TransactionServiceImp#shutdown both acquire a lock on shutdownSynchronizer.

Used version:
transactions-osgi:6.0.0:jakarta

Log:

12:48:03.100 [main] INFO  c.a.icatch.imp.ConditionalWaiter - Waiting for condition to become true...
12:48:04.107 [main] WARN  c.a.icatch.imp.TransactionServiceImp - Shutdown; waiting for all active transactions to finish...
12:48:04.107 [main] INFO  c.a.icatch.imp.ConditionalWaiter - Waiting for condition to become true...
12:48:05.112 [main] WARN  c.a.icatch.imp.TransactionServiceImp - Shutdown; waiting for all active transactions to finish...
.... (a lot more of these)

To Reproduce
Steps to reproduce the behavior:

1. Create a custom XA resource and register it in a user transaction
2. Call "commit" on the user transaction
3. Let the resource throw a XAException (XAER_RMERR, XAER_RMFAIL or XA_RETRY)
4. Call "close" on the user transaction

Full example:

import com.atomikos.datasource.ResourceException;
import com.atomikos.datasource.xa.XATransactionalResource;
import com.atomikos.icatch.config.Configuration;
import com.atomikos.icatch.jta.UserTransactionManager;
import jakarta.transaction.*;

import javax.transaction.xa.XAException;
import javax.transaction.xa.XAResource;
import javax.transaction.xa.Xid;
import java.util.concurrent.atomic.AtomicBoolean;

public class Main {
	public static void main(String[] args) throws SystemException, NotSupportedException, RollbackException,
		HeuristicRollbackException, HeuristicMixedException
	{
		final UserTransactionManager utm = new UserTransactionManager();
		utm.init();

		utm.begin();
		try {
			final XAResource resource = new XAResource() {
				final AtomicBoolean failOnFirstCommit = new AtomicBoolean(true);

				@Override
				public void commit(Xid xid, boolean onePhase) throws XAException {
					System.out.println("Entered commit(xid=" + xid + ", onePhase=" + onePhase + ")");

					if (failOnFirstCommit.getAndSet(false)) {
						System.out.println("Failed to commit.");
						throw new XAException(XAException.XAER_RMERR);
					}

					System.out.println("Successfully committed");
				}

				@Override
				public void end(Xid xid, int flags) throws XAException {}

				@Override
				public void forget(Xid xid) throws XAException {}

				@Override
				public int getTransactionTimeout() throws XAException { return 0; }

				@Override
				public boolean isSameRM(XAResource xares) throws XAException { return xares == this; }

				@Override
				public int prepare(Xid xid) throws XAException { return 0; }

				@Override
				public Xid[] recover(int flag) throws XAException { return new Xid[0]; }

				@Override
				public void rollback(Xid xid) throws XAException {}

				@Override
				public boolean setTransactionTimeout(int seconds) throws XAException { return false; }

				@Override
				public void start(Xid xid, int flags) throws XAException {}
			};

			Configuration.addResource(new XATransactionalResource("resource") {
				@Override
				protected XAResource refreshXAConnection() throws ResourceException {
					return resource;
				}
			});

			final Transaction tx = utm.getTransaction();
			tx.enlistResource(resource);
			utm.commit();
		} finally {
			utm.close();
		}
	}
}

Additional context

• Debugging shows, that the method TransactionServiceImp#removeCoordinator ist entered after the successfull commit
• If you change failOnFirstCommit to true in the given example, the transaction is successfully terminated.

I understand that this project is not intended for support - because bug reports may or may not be considered for inclusion some day (in a future release). If this is issue is important to me then I can go to https://www.atomikos.com/Main/SupportOverview and arrange a paid support subscription.