I am using spring boot and hibernate over jpa with tomcat connection pooling. Can you please help me understanding how spring uses DB connections during transactions. For example consider following scenario:
We have DB connection pool of 2 connections.
Spring starts a transaction i.e. call method decorated with #Transactional annotation.
This method do a DB update
The calls an external service
As response is received from the external service, it updates DB and return.
Spring commits the transaction
Assuming the external service(step 4) takes 1 minute to complete, how many DB connections will be available in the DB pool?. Assuming, spring keeps hold of DB connection until the transaction completes, there will be only 1 DB connection available for any request received during this time and if we received more than 1 requests, they will have to wait for DB connection.
Please confirm my understanding and if it is correct, suggest how I can handle this situation in a high transaction volume system.
Thanks
First your understanding is correct. See the spring documentation about declarative transaction management.
I guess you do the external service call within the transaction, because you want the database changes to be rollbacked in case of an exception. Or in other words you want the db updates to reflect the state of the external service call.
If so you can't move it out the transaction boundary. In this case you should either increase your connection pool size or maybe you can delegate long running transactions to a dedicated server node that handles them. This would keep e.g. a "main" server node that handles user requests away from long running transactions.
And you should think about the data consistency. Is it really necessary that the db update must be synchronized with the external service call? Can the external service call be moved out of the transaction boundary?
You can specify initial size and maximum size of connection pool as per your requirement(depends on the performance of your application).
For example,
<bean id="springDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close" >
<property name="url" value="jdbc:oracle:thin:#localhost:1521:SPRING_TEST" />
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<property name="username" value="root" />
<property name="password" value="root" />
<property name="removeAbandoned" value="true"/>
<property name="initialSize" value="20" />
<property name="maxActive" value="30" />
</bean>
this will create 20 database connection as initialSize is 20 and goes up to 30 Database connection if required as maxActive is 30. you can customize your database connection pool by using different properties provided by Apache DBCP library. Above example is creating connection pool with Oracle 11g database and i am using oracle.jdbc.driver.OracleDriver comes along with ojdbc6.jar or ojdbc6_g.jar
Related
I have implemeted a web application using Struts2 and Hibernate 4.3.5
I am facing some problems and need some clarification on some connection pooling properties.
To make connection pooling with hibernate I have used c3p0-0.9.1.jar
Below are the connection pooling Properties of c3p0 in hibernate.cfg.xml
<!-- Connection Pooling using c3p0 -->
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">10</property>
<property name="hibernate.c3p0.timeout">500</property>
<property name="hibernate.c3p0.max_statements">2</property>// I heard somewhere that this property improves performance.. But dont know How it improves
<property name="hibernate.c3p0.idle_test_period">300</property>
<property name="hibernate.c3p0.preferredTestQuery">
select sysdate from dual;
</property>
In my application I have used * session-per-request* patterns by setting following property in hibernte-cfg.xml
<!-- To open the connection when the request hits. -->
<property name="hibernate.current_session_context_class">thread</property>
When I use this I am getting Session as following
//Here MyHiberantaeUtil is my own class having singleton pattern and Static block
// To read hibernate-cfg.xml and will create SessionFactroy object.
SessionFactory sessionFactory = MyHibernateUtil.getSessionFactory();
Session session = sessionFactory.getCurrentSession();
As per the documentation .. When I use this pattern connection should be created when request is requested and should be closed when request completed.
For my case Its creating connection .. i.e I can get Session Using this getCurrentSession() But its not closing when The request ended.
To test connection pooling I have used jconsole tool in my jdk
Where I can see
`numBusyConection = 2(If that is two requests.)`
I would like to know is there any thing I missed . Why thread property is not working as expected ?
And how to say that Request has completed and request has generated ?
Is session-per-request is best pattern or better to go any other patterns ?
I did not find this class `org.hibernate.connection.C3P0ConnectionProvider` in any jar files.. But My application is connecting to DB and working fine.
Please help thanks in advance.
The JDBC connection is opened when first needed and closed when the Session is closed. You need to manually close the session when you go out of your Service layer:
session.close();
Otherwise the connections won't be closed and you'll starve the connection pool.
It's a better idea to use Spring, because it has support for automatically creating/closing sessions for you. Your MyHibernateUtil is no match for what Spring has to offer:
local transactions
global transactions
transaction annotations and automatic commit on success or rollback on runtime exceptions
Setting up Spring with Hibernate is as easy as using the HibernateUtil class.
I made a manager/Service Server system.
Manager server collects data from database and send that data to multiple Service servers.
My code works nice when I have only 1 server. My configuration like below.(root-context.xml)
<bean id="connectionFactory"
class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="hostName" value="127.0.0.1"/>
<property name="port" value="6379"/>
</bean>
The Problem is Service servers should be multiple. Is there any way to set multiple connection list with spring configurations? Thanks:D
P.S
I know the way to use JedisHelper.java which can be found easily in github. However, what I want to do is figure this out in spring root-context.xml.
you need to use a Jedis Pool.
http://docs.spring.io/spring-data/redis/docs/1.0.6.RELEASE/api/org/springframework/data/redis/connection/jedis/JedisConnectionFactory.html
You need to specify a pool and set it up with all your servers.
This might be a repetitive question for you, but I couldn't find (atleast I couldn't understand) a satisfactory answer, hence asking again.
I am working with two data sources (MySQL and Oracle). Following is a flow of execution:
Main method-A calls method-B (Which writes into Oracle DB) then it(Method-A) calls method-C (Which writes into mySQL DB) then it(Method-A) calls method-D(Which writes into Oracle DB).
If failure occurs at any of place, everything should be rolled back. Currently only changes in Oracle DB are getting rolled back & mySQL DB is not getting rolled back.
I have defined two transactional managers.
=========> First <=========
<tx:annotation-driven transaction-manager="txManager" mode='proxy' proxy-target-class='true’/>
<bean id="txManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="SessionFactory" />
</bean>
<bean id=“SessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean” parent="AbstractSessionFactory" depends-on="AppConfigHelper”>
<property name="hibernateProperties”>
...
ORACLE DB Properties
</property>
</bean>
<aop:aspectj-autoproxy/>
==============================
=========> Second <=========
<tx:annotation-driven transaction-manager="txManager2" mode='proxy' proxy-target-class='true'/>
<bean id="txManager2" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="SessionFactory2" />
<qualifier value="CherryTransaction" />
</bean>
<aop:aspectj-autoproxy/>
<bean id="SessionFactory2" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean" parent="AbstractSessionFactory2" depends-on="AppConfigHelper">
<property name="hibernateProperties">
...
MYSQL DB Properties
</property>
</bean>
==============================
On top of method-A I have used #Transactional annotation
On top of method-B I have used #Transactional annotation
On top of method-C I have used #Transactional("txManager2")
annotation
On top of method-D I have used #Transactional annotation
Question Is:
Why is MySQL changes are not getting rolled back?
Is the only way to get this working is to use Global transaction management using JTA? (Its a legacy system, and this is the only place where I need to interact with two DBs)
Can you please point me to an example / tutorial where this kind case is handled?
Sincerely thanks for reading this!
For that to work, AFAIK you'd need to use JTA. Even that won't help if you're using a storage engine in MySQL that doesn't support transactions. With MySQL, only InnoDB and BDB storage engines support transactions.
If you are using MySQL with storage engine that supports transactions, you need to configure XA drivers for both Oracle and MySQL datasource and ensure that both datasources are enlisted in the transaction of your container. Spring then needs to participate in the same transaction. You can't use HibernateTransactionManager for this, but need the JtaTransactionManager, as explained in this thread.
So, what you need for this to work is
Use InnoDB or BDB storage engines on MySQL
Configure XA datasources in your application server instead of regular ones
Use JtaTransactionManager on Spring configuration
I'm using JPA (Hibernate as provider), Glassfish and MySQL. Everything works great in development, but when I deploy the app to a test server and let it run (largely idle) overnight, I'm usually greeted with this in the morning:
[#|2011-03-09T15:06:00.229+0000|INFO|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=23;_ThreadName=Thread-1;|ERROR [htt\
p-thread-pool-8080-(1)] (JDBCTransaction.java:91) - JDBC begin failed
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 41,936,868 milliseconds ago. The last packet \
sent successfully to the server was 41,936,868 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expirin\
g and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connec\
tion property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1118)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3321)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1940)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2113)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2562)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4956)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:87)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:60)
I tried using the following in my persistence.xml, but it didn't help:
<property name="hibernate.c3p0.min_size" value="5"/>
<property name="hibernate.c3p0.max_size" value="20"/>
<property name="hibernate.c3p0.idleTestPeriod" value="30"/>
<property name="hibernate.c3p0.timeout" value="0"/>
<property name="hibernate.c3p0.max_statements" value="0"/>
So that's the C3p0 configuration; it's entirely possible I'm missing the part that actually tells hibernate "hey, use c3p0".
I'm about to try the suggestion that's right there in the error message: add autoReconnect=true to my JDBC URL, but this is really starting to feel like cargo-cult development at this point. I would appreciate some guidance on the proper way to address this issue. It's hard to debug, because the test cycle is effectively "run it overnight, see what happens in the morning".
I should probably mention how I'm actually using connections in my app. I have a custom Servlet Filter that intercepts all requests. It creates an EntityManager, stores it in a ThreadLocal, and is closed by the filter in a catch/finally block. All my entities obtain a reference to the EntityManager from the ThreadLocal.
It's entirely possible that my filter is at fault, but as it only seems to happen after idle periods, I suspect something else is wrong. I do intend to move to Seam/Weld when I have a chance to catch my breath, but for now I'm relying on this filter.
Edit: here's the TL;DR solution:
use your container's connection pool, if you can (thanks, #partenon)
make sure your connection pool uses connection validation (thanks, #matt b)
In my case, I had to go into the Glassfish console under Resources/JDBC/Connection Pools, Advanced Tab, and then enable Connection Validation:
This was really the crucial step. You also probably want to set Validate At Most Once to something reasonable, say 100 seconds. If you're using C3P0 or similar, make sure you configure idle_test_period and preferredTestQuery.
Whatever you end up doing, it's important to test out your changes to see if they have the desired effect. To make the timeout happen faster in MySQL, you can temporarily set the wait_timeout to something low like 30 seconds by editing my.cnf. This was a tremendous help in debugging this problem, as it allowed me to test changes in seconds, rather than hours.
I think the real question is: why are you using an external connection pooling mechanism instead of using Glassfish' own pooling? Your app server is better suited to provide this kind of service to your application. "External" connection pooling mechanisms are better suited for standalone applications, not in-container applications.
You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
Just a shot in the dark, but have you taken a look at setting the autoReconnect=true property in your JDBC driver? Or consider disabling the server-side setting for client connection timeouts.
I think the property to set the connection test period is idle_test_period, not idleTestPeriod as per C3P0 documentation here. So you should be using:
<property name="hibernate.c3p0.idle_test_period" value="30"/>
instead.
I was getting the same problem and it took time to figure out the solution.
I use Hibernate 4.0.1 and mysql 5.1(no spring framework) and I was facing the issue. First make sure that you configured the c3p0 jars properly which are essential.
I used these properties in hibernate.cfg.xml
<property name="hibernate.c3p0.validate">true</property>
<property name="hibernate.connection.provider_class">org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.preferredTestQuery">SELECT 1;</property>
<property name="hibernate.c3p0.testConnectionOnCheckout">true</property>
<property name="hibernate.c3p0.idle_test_period">10</property>
<property name="hibernate.c3p0.acquireRetryAttempts">5</property>
<property name="hibernate.c3p0.acquireRetryDelay">200</property>
<property name="hibernate.c3p0.timeout">40</property>
But it's of no use 'cause C3p0 was still taking the default properties not the properties which I set in hibernate.cfg.xml, You can check it in logs. So, I searched many websites for right solution and finally I came up with this. remove the C3p0 properties in cfg.xml and create c3p0-config.xml in the root path(along with cfg.xml) and set properties as follows.
<c3p0-config>
<default-config>
<property name="automaticTestTable">con_test</property>
<property name="checkoutTimeout">40</property>
<property name="idleConnectionTestPeriod">10</property>
<property name="initialPoolSize">10</property>
<property name="maxPoolSize">20</property>
<property name="minPoolSize">5</property>
<property name="maxStatements">50</property>
<property name="preferredTestQuery">SELECT 1;</property>
<property name="acquireRetryAttempts">5</property>
<property name="acquireRetryDelay">200</property>
<property name="maxIdleTime">30</property>
</default-config>
</c3p0-config>
but if you run, ORM takes the jdbc connection but not C3p0 connection pool 'cause we should add these properties in hibernate.cfg.xml
<property name="hibernate.c3p0.validate">true</property>
<property name="hibernate.connection.provider_class">org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider</property>
now everything works fine(At least it worked fine for me) and the issue is solved.
check the following for references.
http://www.mchange.com/projects/c3p0/index.html#configuring_connection_testing
https://community.jboss.org/wiki/HowToConfigureTheC3P0ConnectionPool
I hope this solves your problem.
After reading previous questions about this error, it seems like all of them conclude that you need to enable XA on all of the data sources. But:
What if I don't want a distributed
transaction? What would I do if I want to
start transactions on two different
databases at the same time, but
commit the transaction on one database
and roll back the transaction on
the other?
I'm wondering how my code
actually initiated a distributed
transaction. It looks to me like I'm
starting completely separate
transactions on each of the
databases.
Info about the application:
The application is an EJB running on a Sun Java Application Server 9.1
I use something like the following spring context to set up the hibernate session factories:
<bean id="dbADatasource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="jdbc/dbA"/>
</bean>
<bean id="dbASessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dbADatasource" />
<property name="hibernateProperties">
hibernate.dialect=org.hibernate.dialect.Oracle9Dialect
hibernate.default_schema=schemaA
</property>
<property name="mappingResources">
[mapping resources...]
</property>
</bean>
<bean id="dbBDatasource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="jdbc/dbB"/>
</bean>
<bean id="dbBSessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dbBDatasource" />
<property name="hibernateProperties">
hibernate.dialect=org.hibernate.dialect.Oracle9Dialect
hibernate.default_schema=schemaB
</property>
<property name="mappingResources">
[mapping resources...]
</property>
</bean>
Both of the JNDI resources are javax.sql.ConnectionPoolDatasoure's. They actually both point to the same connection pool, but we have two different JNDI resources because there's the possibility that the two, completely separate, groups of tables will move to different databases in the future.
Then in code, I do:
sessionA = dbASessionFactory.openSession();
sessionB = dbBSessionFactory.openSession();
sessionA.beginTransaction();
sessionB.beginTransaction();
The sessionB.beginTransaction() line produces the error in the title of this post - sometimes. I ran the app on two different sun application servers. On one runs it fine, the other throws the error. I don't see any difference in how the two servers are configured although they do connect to different, but equivalent databases.
So the question is
Why doesn't the above code start
completely independent transactions?
How can I force it to start
independent transactions rather than
a distributed transaction?
What configuration could cause the difference in
behavior between the two application
servers?
Thanks.
P.S. the stack trace is:
Local transaction already has 1 non-XA Resource: cannot add more resources.
at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.enlistResource(J2EETransactionManagerOpt.java:124)
at com.sun.enterprise.resource.ResourceManagerImpl.registerResource(ResourceManagerImpl.java:144)
at com.sun.enterprise.resource.ResourceManagerImpl.enlistResource(ResourceManagerImpl.java:102)
at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:216)
at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:82)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1354)
at [application code ...]
1 Why doesn't the above code start completely independent transactions?
The app. server manages the transaction for you which can, if necessary, be a distributed transaction. It enlists all the participants automatically. When there's only one participant, you don't notice any difference with a plain JDBC transaction, but if there are more than one, a distributed transaction is really needed, hence the error.
2 How can I force it to start independent transactions rather than a
distributed transaction?
You can configure the datasource to be XA or Local. The transactional behavior of Spring/Hibernate can also be configured to use either regular JDBC transactions or delegate the management of transactions to the JTA distributed transaction manager.
I suggest you switch the datasource to non-XA and try to configure Spring/Hibernate to use the JDBC transactions. You should find the relevant information in the documentation, here what I suspect is the line to change:
<bean id="txManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager" />
This should essentially means that you are not using the app. server distributed transaction manager.
3 What configuration could cause the difference in behavior between the
two application servers?
If you have really exactly the same app and configuration, this means that in one case only one participant is enlisted in the dist. transaction, while there are two in the 2nd case. One participant corresponds to one physical connection to a database usually. Could it be that in one case, you use two schema on two different databases, while in the 2nd case you use two schema on the same physical database? A more probable explanation would be that the datasource were configured differently on the two app. server.
PS: If you use JTA distributed transactions, you should use UserTransaction.{begin,commit,rollback} rather than their equivalent on the Session.
After reading previous questions about this error, it seems like all of them conclude that you need to enable XA on all of the data sources.
No, not all, all except one (as the exception is saying) if your application server supports Logging Last Resource (LLR) optimization (which allows to enlist one non-XA resource in a global transaction).
Why doesn't the above code start completely independent transactions?
Because you aren't. When using beginTransaction() behind EJB Session Beans, Hibernate will join the JTA transaction (refer to the documentation for full details). So the first call just works but the second call means enlisting another transactional resource in the current transaction. And since none of your resources are XA, you get an exception.
How can I force it to start independent transactions rather than a distributed transaction?
See #ewernli answer.
What configuration could cause the difference in behavior between the two application servers?
No idea. Maybe one of them is using at least one XA datasource.