Skip to content

DB2 - XA resource prepare for XID - the XA resource has become unavailable raised -7 #240

@hci2

Description

@hci2

Describe the bug
It seems to be quite similar to this issue: #38 and StackOverflow issue: https://stackoverflow.com/questions/49392568/xa-resource-has-become-unavailable-xid-raised-7. So after 9 - 48 hours after starting the application and a long-running distributed transaction, the following exception appears:

2025-07-07 22:34:12.228 DEBUG 26930 --- [nio-8080-exec-1] o.s.t.jta.JtaTransactionManager          : Initiating transaction commit
2025-07-08 06:54:28.012 ERROR 26930 --- [nio-8080-exec-8] c.a.datasource.xa.XAResourceTransaction  : XA resource 'lfvXaDataSource': prepare for XID '6C66722D32363933305F746D313735313931373339393136353030303037:6C66722D32363933305F746D39' raised -7: the XA resource has become unavailable
com.ibm.db2.jcc.am.XaException: [jcc][t4][10401][12066][4.32.28] XA exception: XAER_RMFAIL : [jcc][t4][2030][11211][4.32.28] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: Reply.fill() - insufficient data (-1).  Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001 ERRORCODE=-4228, SQLSTATE=null
	at com.ibm.db2.jcc.am.b7.c(b7.java:561)
	at com.ibm.db2.jcc.am.b7.c(b7.java:590)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:2118)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:1028)
	at com.ibm.db2.jcc.t4.a4.b(a4.java:954)
	at com.ibm.db2.jcc.t4.a4.b(a4.java:949)
	at com.ibm.db2.jcc.t4.a4.prepare(a4.java:938)
	at com.atomikos.datasource.xa.XAResourceTransaction.prepare(XAResourceTransaction.java:384)
	at com.atomikos.icatch.imp.PrepareMessage.send(PrepareMessage.java:40)
	at com.atomikos.icatch.imp.PrepareMessage.send(PrepareMessage.java:19)
	at com.atomikos.icatch.imp.PropagationMessage.submit(PropagationMessage.java:67)
	at com.atomikos.icatch.imp.Propagator$PropagatorThread.run(Propagator.java:63)
	at com.atomikos.icatch.imp.Propagator.submitPropagationMessage(Propagator.java:42)
	at com.atomikos.icatch.imp.ActiveStateHandler.prepare(ActiveStateHandler.java:171)
	at com.atomikos.icatch.imp.CoordinatorImp.prepare(CoordinatorImp.java:523)
	at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:687)
	at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:282)
	at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:172)
	at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:414)
	at com.atomikos.icatch.jta.UserTransactionManager.commit(UserTransactionManager.java:159)
	at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1035)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at at.statistik.lfr.web.extern.AmaResource$$EnhancerBySpringCGLIB$$8a41f5bf.einspielung(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at at.statistik.entis.angular.filter.CspFilter.handle(CspFilter.java:42)
	at at.statistik.entis.angular.filter.CspFilter.doFilter(CspFilter.java:34)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at at.statistik.lfr.config.CorsConfig.doFilter(CorsConfig.java:34)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:355)
	at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:54)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
	at psiprobe.Tomcat90AgentValve.invoke(Tomcat90AgentValve.java:35)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:388)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:936)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.lang.Thread.run(Thread.java:750)
Caused by: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][4.32.28] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: Reply.fill() - insufficient data (-1).  Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
	at com.ibm.db2.jcc.am.b7.a(b7.java:338)
	at com.ibm.db2.jcc.t4.a.a(a.java:577)
	at com.ibm.db2.jcc.t4.a.a(a.java:561)
	at com.ibm.db2.jcc.t4.a.a(a.java:556)
	at com.ibm.db2.jcc.t4.y.b(y.java:315)
	at com.ibm.db2.jcc.t4.y.c(y.java:342)
	at com.ibm.db2.jcc.t4.y.c(y.java:455)
	at com.ibm.db2.jcc.t4.y.v(y.java:1230)
	at com.ibm.db2.jcc.t4.ad.c(ad.java:201)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:2527)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:1012)
	... 111 common frames omitted

It can only be resolved after restarting the whole application. After the first occurrence, it seems to persist. The underlying error of DB2 -4499 is described in IBM Troubleshooting and looks like the database connection was not available. But I checked the database with a db explorer tool and can access it. So I expect that the Atomikos database pooling should do a reconnect like HikariCP in Spring does.

To Reproduce
Steps to reproduce the behavior:

  1. Restart application
  2. Start a long running distributed transaction
  3. Wait and see the error log

Additional context

General setup (used multiple times for different datasources):

@Slf4j
@EnableTransactionManagement
@Configuration
@EnableConfigurationProperties({AtomikosProperties.class, JtaProperties.class})
public class JtaTransactionManagerConfig {

    @Bean(initMethod = "init", destroyMethod = "shutdownWait")
    UserTransactionServiceImp userTransactionService(
            AtomikosProperties atomikosProperties, JtaProperties jtaProperties) {
        Properties properties = new Properties(atomikosProperties.asProperties());
        // This allows you to override spring.jta.atomikos.properties.*
        if (StringUtils.hasText(jtaProperties.getTransactionManagerId())) {
            properties.setProperty("com.atomikos.icatch.tm_unique_name", jtaProperties.getTransactionManagerId());
        }
        if (StringUtils.hasText(jtaProperties.getLogDir())) {
            properties.setProperty("com.atomikos.icatch.log_base_dir", jtaProperties.getLogDir());
        }
        return new UserTransactionServiceImp(properties);
    }

    @Bean(initMethod = "init", destroyMethod = "close")
    @DependsOn("userTransactionService")
    public UserTransactionManager userTransactionManager() {
        UserTransactionManager userTransactionManager = new UserTransactionManager();
        userTransactionManager.setStartupTransactionService(false);
        userTransactionManager.setForceShutdown(true);
        return userTransactionManager;
    }

    @Bean("jtaTransactionManager")
    public JtaTransactionManager jtaTransactionManager(final UserTransactionManager userTransactionManager) {
        JtaTransactionManager jtaTransactionManager = new JtaTransactionManager();
        jtaTransactionManager.setUserTransaction(userTransactionManager); //Bean UserTransaction is not used
        jtaTransactionManager.setTransactionManager(userTransactionManager);
        jtaTransactionManager.setAllowCustomIsolationLevels(true);
        return jtaTransactionManager;
    }

    @Bean
    @Primary
    PlatformTransactionManager transactionManager(final UserTransactionManager userTransactionManager) {
        //Bean UserTransaction is not used
        return new JtaTransactionManager(userTransactionManager, userTransactionManager);
    }

Failing datasource - XA resource:

@Bean
	@Primary
	LocalContainerEntityManagerFactoryBean lfvEntityManagerFactory() {
		HibernateJpaVendorAdapter jpaVendorAdapter = new HibernateJpaVendorAdapter();
		jpaVendorAdapter.setGenerateDdl(false);

		Properties props = new Properties();
		props.put(AvailableSettings.DIALECT, "org.hibernate.dialect.DB2Dialect");
		props.put(AvailableSettings.FORMAT_SQL, config.getPropertiesHibernate().getOrDefault("format_sql", "false"));
		props.put(AvailableSettings.JPA_TRANSACTION_TYPE, "jta");

		LocalContainerEntityManagerFactoryBean factoryBean = new LocalContainerEntityManagerFactoryBean();
		factoryBean.setDataSource(lfvDataSource());
		factoryBean.setJpaVendorAdapter(jpaVendorAdapter);
		factoryBean.setPackagesToScan("...");
		factoryBean.setJpaProperties(props);
		return factoryBean;
	}

	@Bean(name= "dsLfv")
	@DependsOn({
			"lfrConfiguration",
			"jtaTransactionManager",
			"userTransactionManager"
	})
	@Primary
	public DataSource lfvDataSource() {
		log.debug("lfvDataSource()");
		final URI jdbcUri = URI.create(config.getJdbcUrl().trim().substring(5)); // Substring to skip prefix "jdbc:"

		DB2XADataSource xaDataSource = new DB2XADataSource();
		xaDataSource.setDatabaseName(jdbcUri.getPath().substring(1, 6));
		xaDataSource.setCurrentSchema(config.getSchemaLfrv().trim());
		xaDataSource.setUser(config.getJdbcUsername().trim());
		xaDataSource.setPassword(config.getJdbcPassword().trim());
		xaDataSource.setSslConnection(true);
//		This leads to InvalidAlgorithmParameterException: DH key size must be multiple of 64, and can only range from 512 to 8192 (inclusive). The specific key size 256 is not supported
//		xaDataSource.setSecurityMechanism(DB2BaseDataSource.ENCRYPTED_USER_AND_PASSWORD_SECURITY);
		xaDataSource.setServerName(jdbcUri.getHost());
		xaDataSource.setPortNumber(jdbcUri.getPort());
		xaDataSource.setDriverType(4); //Thin driver type used for JDBC, see https://www.atomikos.com/Documentation/ConfiguringDB2

		AtomikosDataSourceBean atomikosDataSource = new AtomikosDataSourceBean();
		atomikosDataSource.setXaDataSource(xaDataSource);
		atomikosDataSource.setUniqueResourceName("...");
		atomikosDataSource.setPoolSize(3);
		atomikosDataSource.setMaxPoolSize(20);
		atomikosDataSource.setMaxIdleTime(600000);
		atomikosDataSource.setMaxLifetime(1800000);
		atomikosDataSource.setBorrowConnectionTimeout(30000);
		atomikosDataSource.setTestQuery("SELECT current date FROM sysibm.sysdummy1");
		return atomikosDataSource;
	}

	@Bean(name = "jdbcLfv")
	@Autowired
	public JdbcTemplate lfvJdbcTemplate(@Qualifier("dsLfv") DataSource dsLfv) {
		return new JdbcTemplate(dsLfv);
	}

Tooling:

  • Spring Boot: 2.7.18
  • Java: 8
  • Maven
  • spring-boot-starter-jta-atomikos: 2.7.18
  • atomikos: 4.0.6
  • App Server: Tomcat
  • OS: Linux

I understand that this project is not intended for support - because bug reports may or may not be considered for inclusion some day (in a future release). If this is issue is important to me then I can go to https://www.atomikos.com/Main/SupportOverview and arrange a paid support subscription.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions