We Keep Coding

How rollback transaction after timeout in spring boot application in same way as on weblogic

So in my weblogic application we are you using some jtaWeblogicTransactionManager. There is some default timeout which can be override in annotation #Transactional(timeout = 60). I created some infinity loop to read data from db which correctly timeout:
29 Apr 2018 20:44:55,458 WARN [[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'] org.springframework.jdbc.support.SQLErrorCodesFactory : Error while extracting database name - falli
ng back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLExceptio
n: Transaction rolled back: Transaction timed out after 240 seconds
BEA1-2C705D7476A3E21D0AB1
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1760)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1645)
at weblogic.jdbc.wrapper.JTAConnection.getXAConn(JTAConnection.java:232)
at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:94)
at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:77)
at weblogic.jdbc.wrapper.Connection.preInvocationHandler(Connection.java:107)
at weblogic.jdbc.wrapper.Connection.getMetaData(Connection.java:560)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:331)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:212)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:134)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:97)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:655)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:690)
now I would like to make same behavior in my spring boot application so I tried this:
#EnableTransactionManagement
.
.
.
#Bean(name = "ds1")
#ConfigurationProperties(prefix = "datasource.ds1")
public DataSource logDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
return ds;
}
#Bean(name = "ds2")
#ConfigurationProperties(prefix = "datasource.ds2")
public DataSource refDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
return ds;
}
tm:
#Bean(name = "userTransaction")
public UserTransaction userTransaction() throws Throwable {
UserTransactionImp userTransactionImp = new UserTransactionImp();
userTransactionImp.setTransactionTimeout(120);
return userTransactionImp;
}
#Bean(name = "atomikosTransactionManager", initMethod = "init", destroyMethod = "close")
public TransactionManager atomikosTransactionManager() throws Throwable {
UserTransactionManager userTransactionManager = new UserTransactionManager();
userTransactionManager.setForceShutdown(false);
userTransactionManager.setTransactionTimeout(120);
return userTransactionManager;
}
#Bean(name = "transactionManager")
#DependsOn({ "userTransaction", "atomikosTransactionManager" })
public JtaTransactionManager transactionManager() throws Throwable {
UserTransaction userTransaction = userTransaction();
TransactionManager atomikosTransactionManager = atomikosTransactionManager();
return new JtaTransactionManager(userTransaction, atomikosTransactionManager);
}
and application.properties:
datasource.ref.xa-data-source-class-name=oracle.jdbc.xa.client.OracleXADataSource
datasource.ref.unique-resource-name=ref
datasource.ref.xa-properties.URL=jdbc:oracle:thin:#...
datasource.ref.xa-properties.user=...
#datasource.ref.xa-properties.databaseName=...
datasource.ref.password=301d24ae7d0d69614734a499df85f1e2
datasource.ref.test-query=SELECT 1 FROM DUAL
datasource.ref.max-pool-size=5
datasource.log.xa-data-source-class-name=oracle.jdbc.xa.client.OracleXADataSource
datasource.log.unique-resource-name=log
datasource.log.xa-properties.URL=jdbc:oracle:thin:#...
datasource.log.xa-properties.user=...
#datasource.log.xa-properties.databaseName=...
datasource.log.password=e58605c2a0b840b7c6d5b20b3692c5db
datasource.log.test-query=SELECT 1 FROM DUAL
datasource.log.max-pool-size=5
spring.jta.atomikos.properties.log-base-dir=target/transaction-logs/
spring.jta.enabled=true
spring.jta.atomikos.properties.service=com.atomikos.icatch.standalone.UserTransactionServiceFactory
spring.jta.atomikos.properties.max-timeout=600000
spring.jta.atomikos.properties.default-jta-timeout=10000
spring.transaction.default-timeout=900
but with no success. My infinity loop never ends (I wait about 15 minutes and then I stop my app). The only time when I saw rollback was when I tried Thread.sleep and after sleep this transaction timeout with rollback but this is not what I want to. So is there some way how to interrupt process after timeout(use timeout in annotation or use default) in same way how in my weblogic application ?
UPDATE
I tested it like this:
public class MyService {
public void customMethod(){
customDao.readSomething();
}
}
public class CustomDao {
#Transactional(timeout = 120)
public void readSomething()
while(true){
//read data from db. app on weblogic throw timeout, spring boot app in docker did nothing and after 15 I give it up and kill it
}
}
}
UPDATE2
When I turn on atomikos debug I can see there is warning during init and some atomikos timer:
2018-05-03 14:00:54.833 [main] WARN c.a.r.xa.XaResourceRecoveryManager - Error while retrieving xids from resource - will retry later...
javax.transaction.xa.XAException: null
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:730)
at com.atomikos.datasource.xa.RecoveryScan.recoverXids(RecoveryScan.java:32)
at com.atomikos.recovery.xa.XaResourceRecoveryManager.retrievePreparedXidsFromXaResource(XaResourceRecoveryManager.java:158)
at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67)
at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:449)
at com.atomikos.datasource.xa.XATransactionalResource.setRecoveryService(XATransactionalResource.java:416)
at com.atomikos.icatch.config.Configuration.notifyAfterInit(Configuration.java:466)
at com.atomikos.icatch.config.Configuration.init(Configuration.java:450)
at com.atomikos.icatch.config.UserTransactionServiceImp.initialize(UserTransactionServiceImp.java:105)
at com.atomikos.icatch.config.UserTransactionServiceImp.init(UserTransactionServiceImp.java:219)
at com.atomikos.icatch.jta.UserTransactionImp.checkSetup(UserTransactionImp.java:59)
at com.atomikos.icatch.jta.UserTransactionImp.setTransactionTimeout(UserTransactionImp.java:127)
maybe this is the reason. How I can fix this ? I am using oracle 12 with ojdbc8 driver
UPDATE 3
after fix UPDATE2 to grant user permission to db I can see in log warning:
2018-05-03 15:16:30.207 [Atomikos:4] WARN c.a.icatch.imp.ActiveStateHandler - Transaction 127.0.1.1.tm152535336001600001 has timed out and will rollback.
problem is that app is still reading data from db after this timeout. Why it is not rollbacked ?
UPDATE 4
so I found in ActiveStateHandler when timeout occurs there is code:
...
setState ( TxState.ACTIVE );
...
and AtomikosConnectionProxy is checking timeout this way
if ( ct.getState().equals(TxState.ACTIVE) ) ct.registerSynchronization(new JdbcRequeueSynchronization( this , ct ));
else AtomikosSQLException.throwAtomikosSQLException("The transaction has timed out - try increasing the timeout if needed");
so why timeout is set state which not cause exception in AtomikosConnectionProxy ?
UPDATE 5
so I found that property
com.atomikos.icatch.threaded_2pc
will solve my problem and now it starts rollback how I want. But I still dont understand why I should set this to true because now I am testing it on some task which should run in single thread

set com.atomikos.icatch.threaded_2pc=true in jta.properties fixed my problem. Idk why this default value was change to false in web application.
* #param single_threaded_2pc (!com.atomikos.icatch.threaded_2pc)
* If true then commit is done in the same thread as the one that
* started the tx.

XA transactions are horribly complicated and you really want to have a very good reason for using them (ie it's literally impossible to add some business process that removes the need for XA), because you are going to get into trouble out in the wild...
That said, My guess is that it's about timeout discrepancies between XA phases.
With XA there are 2 timeouts - a timeout for the 1st phase, known as the Voting phase (which is typically the one set by the #Transactional annotation, but this depends on the JTA provider) and another timeout for the 2nd phase, known as the commit phase, which is typically a lot longer, because the Transaction Manager has already got the agreement from all parties that the commit is ready to go, and therefore provide greater leeway for things like transient network failures and so on.
My guess is that the WebLogic JTA is simply behaving differently to Atomikos with how it's handling the 2nd phase notifications back from the participants, until atomikos is changed to use the multithreaded ack.
If you application is just you and the database, then you can probably get away without an XA Transaction Manager. I'd expect this would behave the way you want for timeouts.
Good Luck!

Quartz 2.2.3 JobStore Property Being Overriden by Default Settings - Spring Boot, Liquibase, Oracle

I am trying to implement a Quartz scheduler in my Spring Boot application; however, I continue to receive the following error after startup:
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "SCHED_NAME": invalid identifier.
In the startup logs before the error, I see:
liquibase : Successfully acquired change log lock
liquibase : Reading from PARTSVOICE_APP.DATABASECHANGELOG
liquibase : Successfully released change log lock
o.q.i.StdSchedulerFactory : Using default implementation for ThreadExecutor
o.q.c.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
o.q.c.QuartzScheduler : Quartz Scheduler v.2.2.3 created.
o.s.s.q.LocalDataSourceJobStore : Using db table-based data access locking (synchronization).
o.s.s.q.LocalDataSourceJobStore : JobStoreCMT initialized.
o.q.c.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.2.3) 'scheduler' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 25 threads.
Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is not clustered.
This message is concerning because in my application.yml, my setup is the following:
org:
quartz:
scheduler:
instanceName: cdp-scheduler
instanceId: AUTO
threadPool:
threadCount: 25
class: org.quartz.simpl.SimpleThreadPool
jobStore:
class: org.quartz.impl.jdbcjobstore.JobStoreTX
tablePrefix: c_
I read these properties and put them within a SchedulerFactoryBean here:
#Bean
public SchedulerFactoryBean scheduler(DataSource ds, SpringLiquibase dependent) throws SchedulerException, ConnectionException {
SchedulerFactoryBean factory = new SchedulerFactoryBean();
Properties quartzProps = new Properties();
quartzProps.setProperty("org.quartz.scheduler.instanceId", quartzConfig.getInstanceId());
quartzProps.setProperty("org.quartz.scheduler.instanceName", quartzConfig.getInstanceName());
quartzProps.setProperty("org.quartz.threadPool.threadCount", quartzConfig.getThreadCount());
quartzProps.setProperty("org.quartz.jobStore.class", quartzConfig.getJobStoreClass());
quartzProps.setProperty("org.quartz.jobStore.driverDelegateClass", "org.quartz.impl.jdbcjobstore.oracle.OracleDelegate");
quartzProps.setProperty("org.quartz.threadPool.class", quartzConfig.getThreadPoolClass());
factory.setOverwriteExistingJobs(true);
factory.setAutoStartup(false);
factory.setDataSource(ds);
factory.setQuartzProperties(quartzProps);
factory.setExposeSchedulerInRepository(true);
factory.setAutoStartup(true);
AutowiringSpringBeanJobFactory jobFactory = new AutowiringSpringBeanJobFactory();
jobFactory.setApplicationContext(applicationContext);
factory.setJobFactory(jobFactory);
return factory;
}
I am trying to use the org.quartz.impl.jdbcjobstore.JobStoreTX class so that I can store the Quartz information in our database. However, I believe this is getting overridden from what I can see from the log message above. A quick Google search has shown me that if you provide a datasource, then the JobStoreCMT class is automatically implemented but that doesn't make sense to me.
I'm also using Liquibase to execute a SQL script that creates the quartz tables, this is being executed fine, it drops and creates the tables I need. I've also tried using Flybase but I get the same error, so it definitely is related to my quartz setup.
Has anybody had a similar experience? Any suggestions? Please let me know if you think I should supply more information. Thanks.

For the error I can infer that the tables aren't properly created (either not at all or using the wrong scripts). For Quartz, you need to create the tables using the db scripts for you database type, which you can find in the quartz download.
Here is a similar case: https://groups.google.com/forum/#!topic/axonframework/IlWZ0UHK2hk
If you think that the script is OK, please try to create the table directly in your database.
In case that it still does not work you can go to the next step and check the configuration. Maybe the tablePrefix that you are using is not well interpreted. Try to use the default QRTZ_.
Here is an setup example with a lower version of Quartz.
http://www.opencodez.com/java/quartz-scheduler-with-spring-boot.htm
Let me know if it works!

Max number of request parameters reached in spring boot application

I have a spring boot application that's using multi-upload to update sometimes large amounts of files 10K+. In those cases, I'm hitting this exception. I'm guessing it's looking at my "files" parameter and seeing that it's an array > 10K and flagging this exception. I'm also sending another parameter that's an array of strings that are associated with the list of files, its size being the number of files, > 10K
java.lang.IllegalStateException: More than the maximum number of request parameters (GET plus POST) for a single request ([10,000]) were detected. Any parameters beyond this limit have been ignored. To change this limit, set the maxParameterCount attribute on the Connector.
at org.apache.tomcat.util.http.Parameters.addParameter(Parameters.java:204) ~[tomcat-embed-core-8.5.11.jar:8.5.11]
at org.apache.catalina.connector.Request.parseParts(Request.java:2860) ~[tomcat-embed-core-8.5.11.jar:8.5.11]
at org.apache.catalina.connector.Request.parseParameters(Request.java:3177) ~[tomcat-embed-core-8.5.11.jar:8.5.11]
at org.apache.catalina.connector.Request.getParameter(Request.java:1110) ~[tomcat-embed-core-8.5.11.jar:8.5.11]
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381) ~[tomcat-embed-core-8.5.11.jar:8.5.11]
I understand the exception, but I'm trying to figure out where in my application.properties I can adjust this. I've set spring.http.multipart.max-file-size and spring.http.multipart.max-request-size there. I'm not finding anything equivilant to the maxParameterCount in this source.
Also, assuming there's a way I can set it for the instance running locally with spring boot (tomcat embedded), will the change also work in the deploymenet environment, or does that require changing a tomcat configuration?
Update: I found a solution that works when running locally with spring boot. I assume since this is changing the Tomcat Embedded instance, that this wouldn't apply to a deployed full tomcat environment -- I am wondering if there's a solution that would work in both tomcat instances.
#Configuration
public class TomcatCustomizationConfiguration {
#Bean
public TomcatEmbeddedServletContainerFactory tomcatEmbeddedServletContainerFactory() {
final int maxHttpRequests = 50000;
TomcatEmbeddedServletContainerFactory tomcatFactory = new TomcatEmbeddedServletContainerFactory();
tomcatFactory.addConnectorCustomizers(connector -> connector.setMaxParameterCount(maxHttpRequests));
return tomcatFactory;
}
}

According to the Spring documentation, you can add missing configuration yourself by extending the WebServerFactoryCustomizer:
If a configuration key doesn’t exist for your use case, you should then look at WebServerFactoryCustomizer. You can declare such a component and get access to the server factory and the chosen web stack.
As there is no server.tomcat.max-parameter-count configuration yet, you can add it kind of the same way as OPs configuration code:
#Configuration
public class TomcatCustomizationConfiguration implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {
#Value("${server.tomcat.max-parameter-count:10000}")
private int maxParameterCount;
#Override
public void customize(TomcatServletWebServerFactory factory) {
factory.addConnectorCustomizers(connector -> connector.setMaxParameterCount(maxParameterCount));
}
}
N.B. I actually found the solution at this blog.

Cannot undeploy from Tomcat due to specific Spring JMS configuration

I have used ActiveMQ as JMS implementation (activemq-spring 5.12.1) and Spring JMS integration (spring-jms 4.2.3.RELEASE), all wrapped in Spring Boot web application, being deployed on Tomcat.
I have following Spring configuration (code reduced for the verbosity of code sample):
#Configuration
#EnableJms
public class AppConfiguration {
#Bean
public XAConnectionFactory jmsXaConnection(String activeMqUsername, String activeMqPassword) {
ActiveMQXAConnectionFactory activeMQXAConnectionFactory = new ActiveMQXAConnectionFactory(activeMqUsername, activeMqPassword, activeMqUrl);
ActiveMQPrefetchPolicy prefetchPolicy = new ActiveMQPrefetchPolicy();
prefetchPolicy.setAll(0);
activeMQXAConnectionFactory.setPrefetchPolicy(prefetchPolicy);
return activeMQXAConnectionFactory;
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory, JtaTransactionManager jtaTransactionManager) {
DefaultJmsListenerContainerFactory containerFactory = new DefaultJmsListenerContainerFactory();
containerFactory.setConnectionFactory(connectionFactory);
containerFactory.setTransactionManager(jtaTransactionManager);
containerFactory.setSessionTransacted(true);
containerFactory.setTaskExecutor(Executors.newFixedThreadPool(2));
containerFactory.setConcurrency("2-2");
containerFactory.setCacheLevel(DefaultMessageListenerContainer.CACHE_CONSUMER);
return containerFactory;
}
}
My target was to configure two consumers (hence concurrecny set to 2-2) and to prevent any messages caching (hence prefetch policy set to 0).
It works, but causes very unpleasent side effect:
When I try to undeploy the application via Tomcat Manager, it hangs for a while and then indefinitely, every second produces following DEBUG message:
"DefaultMessageListenerContainer:563 - Still waiting for shutdown of 2 Message listener invokers".
Therefore, I am forced to kill Tomcat process every time. What have I done wrong?
One of my lucky shots (documentation both ActiveMQ and Spring JMS was not that helpful), was to set prefetch policy to 1 instead of 0. Then it undeploys gracefully, but I cannot see how it can relate.
Also I am curious, why having cache level set to CACHE_CONSUMER is required for the ActiveMQ to create two consumers. When default setting was left (CACHE_NONE while using external transaction manager), only one consumer was created (while concurrency was still set two 2-2, and so was TaskExecutor).
If it matters, for connection factory and transaction manager, Atomikos is used. I can paste its configuration also, but it seems irrelevant.

Most likely this means the consumer threads are "stuck" in user code; take a thread dump with jstack to see what the container threads are doing.

Client's Transaction Aborted when accessing two databases

I have a propertiesbean which gives me SqlSessions.
This bean is annotated with all this jazz # the class level
#Singleton
#LocalBean
#Startup
#TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)
i also have this
public SqlSessionFactory getSqlSessionFactory(ConnTypes connType) {...}
which returns the SqlSession you want.
I have two databases. One is a mysql instance and the other one isn't (lets call it db2).
When I run the project locally everything is fine. Both databases are accessed with no problem.
When I run the project on our test server it starts to throw a Client Transaction aborted error. I've done a fair amount of research on this and it seems like people get these exceptions when there is a problem with a database query or some database access. The entire transaction is then marked as rollback and this exception is thrown (at least that's what i've read)
It looks to me like it throws an XA-Resource error right before the first time it throws the client's transaction aborted error. I know that you can get those when your bean tries to access another session. I've seen this error before when running locally and trying to maintain connections to both databases in one method.
Could it be that my singleton properties bean, which accesses both databases, gets into a weird transactional state trying to return a mysql session in one thread and a db2 session in the other?