Crash on an interrupted thread

[miroch41]

Bug Description
If a thread running the transaction gets interrupted, Atomikos fails with a crash.

To reproduce
Just call

Thread.currentThread().interrupt();

On transaction boundary the application crashes, as Atomikos tries to write a checkpoint. However, writing in the respective file fails in the interrupted thread. If you try to commit, it fails and data are not committed. The exception follows:

Caused by: jakarta.transaction.RollbackException: Error in committing: could not flush state image java.nio.channels.ClosedByInterruptException com.atomikos.recovery.LogWriteException - recovery will clean up in the background
	at com.atomikos.icatch.jta.TransactionImp.rethrowAsJtaRollbackException(TransactionImp.java:49)
	at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:192)
	at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:428)
	at com.atomikos.icatch.jta.UserTransactionManager.commit(UserTransactionManager.java:160)
	at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1026)
	... 44 more
Caused by: java.lang.IllegalStateException: could not flush state image java.nio.channels.ClosedByInterruptException com.atomikos.recovery.LogWriteException
	at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:54)
	at com.atomikos.finitestates.FSMImp.notifyListeners(FSMImp.java:102)
	at com.atomikos.finitestates.FSMImp.setState(FSMImp.java:180)
	at com.atomikos.icatch.imp.CoordinatorImp.setState(CoordinatorImp.java:283)
	at com.atomikos.icatch.imp.CoordinatorStateHandler.commitFromWithinCallback(CoordinatorStateHandler.java:346)
	at com.atomikos.icatch.imp.ActiveStateHandler$6.doCommit(ActiveStateHandler.java:285)
	at com.atomikos.icatch.imp.CoordinatorStateHandler.commitWithAfterCompletionNotification(CoordinatorStateHandler.java:587)
	at com.atomikos.icatch.imp.ActiveStateHandler.commit(ActiveStateHandler.java:280)
	at com.atomikos.icatch.imp.CoordinatorImp.commit(CoordinatorImp.java:547)
	at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:679)
	at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:279)
	at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:178)
	... 47 more
Caused by: com.atomikos.recovery.LogWriteException: java.nio.channels.ClosedByInterruptException
	at com.atomikos.recovery.fs.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:203)
	at com.atomikos.recovery.fs.CachedRepository.performCheckpoint(CachedRepository.java:84)
	at com.atomikos.recovery.fs.CachedRepository.put(CachedRepository.java:77)
	at com.atomikos.recovery.fs.OltpLogImp.write(OltpLogImp.java:46)
	at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:51)
	... 58 more
Caused by: java.nio.channels.ClosedByInterruptException
	at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200)
	at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172)
	at java.base/sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:532)
	at com.atomikos.recovery.fs.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:196)
	... 62 more

If you try to rollback, crash occurs either. The exception follows:

Caused by: com.atomikos.icatch.jta.ExtendedSystemException: Error in rollback: could not flush state image java.nio.channels.ClosedByInterruptException com.atomikos.recovery.LogWriteException
	at com.atomikos.icatch.jta.TransactionImp.rollback(TransactionImp.java:206)
	at com.atomikos.icatch.jta.TransactionManagerImp.rollback(TransactionManagerImp.java:440)
	at com.atomikos.icatch.jta.UserTransactionManager.rollback(UserTransactionManager.java:199)
	at org.springframework.transaction.jta.JtaTransactionManager.doRollback(JtaTransactionManager.java:1053)
	... 44 more
Caused by: com.atomikos.icatch.SysException: Error in rollback: could not flush state image java.nio.channels.ClosedByInterruptException com.atomikos.recovery.LogWriteException
	at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackFromWithinCallback(CoordinatorStateHandler.java:518)
	at com.atomikos.icatch.imp.ActiveStateHandler$7.doRollback(ActiveStateHandler.java:302)
	at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackWithAfterCompletionNotification(CoordinatorStateHandler.java:569)
	at com.atomikos.icatch.imp.ActiveStateHandler.rollback(ActiveStateHandler.java:297)
	at com.atomikos.icatch.imp.CoordinatorImp.rollback(CoordinatorImp.java:576)
	at com.atomikos.icatch.imp.TransactionStateHandler.rollback(TransactionStateHandler.java:171)
	at com.atomikos.icatch.imp.TransactionStateHandler.rollbackWithStateCheck(TransactionStateHandler.java:186)
	at com.atomikos.icatch.imp.CompositeTransactionImp.doRollback(CompositeTransactionImp.java:193)
	at com.atomikos.icatch.imp.CompositeTransactionImp.rollback(CompositeTransactionImp.java:317)
	at com.atomikos.icatch.jta.TransactionImp.rollback(TransactionImp.java:203)
	... 47 more
Caused by: java.lang.IllegalStateException: could not flush state image java.nio.channels.ClosedByInterruptException com.atomikos.recovery.LogWriteException
	at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:54)
	at com.atomikos.finitestates.FSMImp.notifyListeners(FSMImp.java:102)
	at com.atomikos.finitestates.FSMImp.setState(FSMImp.java:180)
	at com.atomikos.icatch.imp.CoordinatorImp.setState(CoordinatorImp.java:283)
	at com.atomikos.icatch.imp.CoordinatorImp.setStateHandler(CoordinatorImp.java:202)
	at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackFromWithinCallback(CoordinatorStateHandler.java:512)
	... 56 more
Caused by: com.atomikos.recovery.LogWriteException: java.nio.channels.ClosedByInterruptException
	at com.atomikos.recovery.fs.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:203)
	at com.atomikos.recovery.fs.CachedRepository.performCheckpoint(CachedRepository.java:84)
	at com.atomikos.recovery.fs.CachedRepository.put(CachedRepository.java:77)
	at com.atomikos.recovery.fs.OltpLogImp.write(OltpLogImp.java:46)
	at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:51)
	... 61 more
Caused by: java.nio.channels.ClosedByInterruptException
	at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200)
	at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172)
	at java.base/sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:532)
	at com.atomikos.recovery.fs.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:196)
	... 65 more

TransactionsEssentials version
6.0.0

[henrik42]

Connection Leak - pool gets exhausted

We're using the fork https://github.com/aubelix/transactions-essentials and also ran into this.

We're delegating tasks to a pool of threads, the main thread "joins" them by consuming a Future for each task. Tasks use read-only transactions and run some queries to collect data.

When the joining main thread receives an exception because the joined task threw, it will send Thread.interrupt() to all tasks/threads running (to make them terminate asap).

This causes the java.nio.channels.ClosedByInterruptException that was reported in this issue.

We found, that after this happens, the connection of the throwing thread will not be "available" in the pool.

I believe, there is a bug in com.atomikos.icatch.imp.CoordinatorStateHandler::rollbackWithAfterCompletionNotification.

The exception will come as com.atomikos.icatch.SysException: Error in rollback:... out of cb.doRollback() so none of the catch-clauses will run. So no notification takes place and the connection will not be set to TERMINATED.

		try {
        	cb.doRollback();
        	coordinator_.notifySynchronizationsAfterCompletion(TxState.ABORTING, TxState.TERMINATED);
    	} catch (HeurCommitException hc) {
    		coordinator_.notifySynchronizationsAfterCompletion(TxState.COMMITTING, TxState.TERMINATED);
    		throw hc;
    	} catch (HeurMixedException hm) {
    		coordinator_.notifySynchronizationsAfterCompletion(TxState.ABORTING, TxState.TERMINATED);
    		throw hm;
    	} catch (HeurHazardException hh) {
    		coordinator_.notifySynchronizationsAfterCompletion(TxState.ABORTING,TxState.TERMINATED);
    		throw hh;
    	}
	}

We try to fix this by adding:

    	} catch (RuntimeException rtx) {
            coordinator_.notifySynchronizationsAfterCompletion(TxState.ABORTING,TxState.TERMINATED);
            throw rtx;
        } catch (Throwable t) {
            coordinator_.notifySynchronizationsAfterCompletion(TxState.ABORTING,TxState.TERMINATED);
            throw new RuntimeException("CoordinatorStateHandler: Oops : " + t, t);
        }

With this fix the connection gets returned to the pool.

I'm a complete Atomikos noob, so I'd be really glad if someone could say something to this fix and maybe point to other places that would need to be fixed also.

Cheers Henrik