I'm using spring DMLC for my application with below settings, i'm facing strange behavior with DMLC that if I send 1000 messages on listener queue only ~1990 reaches to dmlc very quickly and ~10 get stuck on server, on further analysis i found that acknowledgements are not sent back for those 10 that's why i can see them on server, after few minutes acks is sent back but very slowly.
further on this i tried cacheConsumers=false in CachingConnectionFactory and everything becomes fine, however this makes frequent bind/unbind to mq server and creates huge consumer objects in jmv, does anyone have any solution how to solve this issue keeping cacheConsumers=true ?
<bean id="listenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="cachingjmsQueueConnectionFactory" />
<property name="destination" ref="queueDestination" />
<property name="messageListener" ref="queueDestination" />
<property name="concurrency" value="10-10" />
<property name="cacheLevel" value="1" />
<property name="transactionManager" ref="dbTransactionManager" />
<property name="sessionTransacted" value="true" />
</bean>
<bean id="cachingjmsQueueConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="jmsQueueConnectionFactory" />
<property name="reconnectOnException" value="true" />
<property name="cacheConsumers" value="true" />
<property name="cacheProducers" value="true" />
<property name="sessionCacheSize" value="1" />
</bean>
You can set cacheConsumer to false on the cachingConnectionFactory and also change the cacheLevel to level 3 (CACHE_CONSUMER) on the DefaultMessageListenerClass. This way, the consumer will be cached at the DMLC level and the issue with stuck messages should be resolved without seeing frequent binds/unbinds.
The cacheConsumer should be set to false and you should have the DefaultMessageListenerClasse control the caching because is preferable to have the listener container handle appropriate caching within it's lifecycle. The following note in the Spring documentation (http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/jms/listener/DefaultMessageListenerContainer.html) discusses this:
Note: Don't use Spring's CachingConnectionFactory in combination with
dynamic scaling. Ideally, don't use it with a message listener
container at all, since it is generally preferable to let the listener
container itself handle appropriate caching within its lifecycle.
Also, stopping and restarting a listener container will only work with
an independent, locally cached Connection - not with an externally
cached one.
Related
I have a Java Spring application connecting to a SQL Server database.
The connection settings are:
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="net.sourceforge.jtds.jdbc.Driver" />
<property name="jdbcUrl"
value="jdbc:jtds:sqlserver://${db.host}:1433/TestDB" />
<property name="user" value="${db.user}" />
<property name="password" value="${db.pass}" />
<!-- these are connection pool properties for C3P0 -->
<property name="minPoolSize" value="10" />
<property name="maxPoolSize" value="100" />
<property name="acquireIncrement" value="5"/>
<property name="maxIdleTime" value="30000" />
</bean>
everything works fine but sometimes I got the following error:
Could not open JDBC Connection for the transaction; nested exception is java.sql.SQLException: I/O Error: Read timed out
I have searched a lot but can't find any clue, any idea or help?
I'm using
<bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
<constructor-arg index="0" ref="sqlSessionFactory" />
</bean>
in my spring-config xml to get my sqlSession, and in the DAO I use:
#Autowired
SqlSession sqlSession;
and then I execute the queries I want. Is it possible that because the connection is not closed that this error exists?
In my case connections were getting dropped when the nightly db backup job triggered. I'm using jtds/sql-server as well. Here is what I did to fix it:
Create/setup a health-check cron job that executes a simple query from within your application, like a short select from. Call it every 10 minutes or so and log the result. It will give you some feedback about when and why this is happening.
Reduce the idle time parameter (maxIdleTime) in your configuration so that old connections get automatically discarded.
Keep in mind that if you don't change the maxIdleTime and you keep multiple connections open, some of them may remain in a bad state even if you are using the health-check function. Quoting from c3p0 documentation:
By default, pools will never expire Connections. If you wish Connections to be expired over time in order to maintain "freshness", set maxIdleTime and/or maxConnectionAge. maxIdleTime defines how many seconds a Connection should be permitted to go unused before being culled from the pool. maxConnectionAge forces the pool to cull any Connections that were acquired from the database more than the set number of seconds in the past.
Another way to setup a health-check call is by using the idleConnectionTestPeriod parameter. Also check this answer which can give you more ideas on how to set it up.
Our application has multiple modules, each module use its own schema in the same mysql database. Now I need to make different connection pool configurations for each module because of their different db resource consuming nature, i.e. some module may have 20 active connections at a point of time, but others may just have 1 max. I have searched here and other forums, couldn't find a solution, just this topic is not about multi-tenancy or multi-database, all schemas are in the same db.
Here's the config we have:
<bean id="dataSource" class="our.own.package.RoutingDataSource"> <!-- RoutingDataSource extends spring AbstractDataSource -->
<property name="master" ref="masterDS"/>
</bean>
<bean id="abstractDataSource" abstract="true">
<property name="driverClass" value="com.mysql.jdbc.Driver" />
<property name="initialPoolSize" value="#initial.pool.size#" />
<property name="minPoolSize" value="#min.pool.size#" />
<property name="maxPoolSize" value="#max.pool.size#" /> <!-- I want to have different configs for each module in our application -->
</bean>
<bean id="masterDS" class="com.mchange.v2.c3p0.ComboPooledDataSource" parent="abstractDataSource">
<property name="jdbcUrl" value="jdbc:mysql://#host#/" />
<property name="user" value="#user#" />
<property name="password" value="#pwd#" />
<property name="dataSourceName" value="#dbName#" />
</bean>
So now I have two questions:
1) Is it possible to have different connection pool configurations for one datasource in Spring?
2) If I have to go with the multiple datasource way(one datasource for one module), is implementing Spring's AbstractRoutingDataSource the correct way to go?
Thank you!
Ad.1 Your data source is in the fact connection pool so you want to have multiple pools on the top of another pool. You can do it but you will face many other problems.
Ad.2. Yes definitely. You already have RoutingDataSource so this one should be implementation of AbstractRoutingDataSource. Probably you already have logic there to determine current the data source routing key which should be used to do the lookup.
I read on : http://www.javaworld.com/article/2074123/java-web-development/transaction-and-redelivery-in-jms.html?page=2
"Generally, acknowledging a particular message acknowledges all prior messages the session receives" ( in Client acknowledgement mode )
"Message redelivery is not automatic, but messages are redelivered under certain circumstances"
My questions :
how can I ensure there is a new session every time I recive a message (but reuse the connection)?
how to enforce the redelivery for un acknowledge message ?
Im using this configration :
<bean id="jmsConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory"
lazy-init="true">
<property name="queueManager" value="${queueManager}" />
<property name="hostName" value="${hostName}" />
<property name="transportType" value="${transportType}" />
<property name="port" value="${port}" />
<property name="channel" value="${channel}" />
<property name="SSLCipherSuite" value="${SSLCipherSuite}" />
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="10"/>
<property name="maximumActive" value="100"/>
<property name="connectionFactory" ref="jmsConnectionFactory"/>
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory"/>
<property name="transacted" value="false"/>
</bean>
<bean id="mqNonJmsDestRes" class="calypsox.tk.util.NonJmsMQQueueDestinationResolver" />
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="configuration" ref="jmsConfig" />
<property name="acknowledgementModeName" value="CLIENT_ACKNOWLEDGE" />
<property name="destinationResolver" ref="mqNonJmsDestRes" />
</bean>
and I use camel processor as endpoint bean as singleton
That article you referenced is from 2002. All of the MQ based systems have received a lot of work since then. On your AMQ PooledConnectionFactory there are settings to control how long your connections last before they are destroyed and what you should do if you encounter an error. I recommend reading into some of the newer documentation since there has been a lot of changes in the last 14 years. So some things have become much easier.
You can also check into the exceptionListener on "org.apache.camel.component.jms.JmsComponent" to configure how to manage exceptions and even write your own if the current options don't suit your needs.
I can't seem to figure this out...
This works perfectly in JAVA:
// Create a Session
Session session = connection.createSession(false, Session.CLIENT_ACKNOWLEDGE); // << means we have to acknowledge that we processed it!
But when I create a Spring Bean, I cannot get the CLIENT_ACKNOWLEDGE to work. It seems that it's auto-acknowledging all messages.
<bean id="testListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer" lazy-init="true">
<property name="connectionFactory" ref="amqConnectionFactory" />
<property name="destinationName" value="MyQueue" />
<property name="messageListener" ref="testListener" />
<property name="autoStartup" value="false"/>
<property name="sessionAcknowledgeModeName" value="CLIENT_ACKNOWLEDGE"/>
<property name="sessionTransacted" value="true"/>
<property name="concurrency" value="2"/>
</bean>
Anything I'm missing?
The DMLC in spring has transactions enabled in your config, while in your Java example, you have transactions disabled. Transactions overrides any acknowledge modes. So, choose if you should go with transactions or client ack since you cannot pick both.
I am using Tomcat JDBC API(org.apache.tomcat.jdbc.pool.DataSource) to connect to my PostgreSQL database from Spring configuration file as shown below. I got a new requirement to configure two databases which should act as a fail over mechanism, Like When one database is down application should automatically switch back to another database.
<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource"
destroy-method="close">
<property name="driverClassName" value="org.postgresql.Driver" />
<property name="url" value="jdbc:postgresql://localhost/dbname?user=postgres" />
<property name="username" value="postgres" />
<property name="password" value="postgres" />
<property name="maxActive" value="5" />
<property name="maxIdle" value="5" />
<property name="minIdle" value="2" />
<property name="initialSize" value="2" />
</bean>
Can anyone suggest how this can be achieved using Spring configuration file.
The normal way this is done is by using virtual IP addresses (with possible forwarding), checking for activity, a shoot-the-other-node-in-the-head approach and proper failover. Spring is exactly the wrong solution to this if you want to avoid things like data loss.
A few recommendations.
repmgr from 2ndquadrant will manage a lot of the process for you.
Use identical hardware and OS and streaming replication.
Use virtual IP addresses, and the like. Use a heartbeat mechanism to trigger failover via rempgr
Then from this perspective your spring app doesn't need reconfiguring.