I have, I might say, a quite a big issue.
I'm working on Java web application which use springs BasicDataSource to setup DB connection. I was testing the application locally and it works just fine... but, when application is online, connection to DB in some point just stuck. I was than investigating regarding connection pooling, and I figure it out that on each new HTTP request, where I have some of the queries executed, new pool is created. As I know, pooling is introduced to be reusable, and not created the each time when new DB access is involved. Or I'm wrong?
Here is my spring datasource config:
<bean id="dataSource" destroy-method="close" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${jdbc.driverClassName}"/>
<property name="url" value="url"/>
<property name="username" value="username"/>
<property name="password" value="password"/>
<property name="defaultAutoCommit" value="true"/>
<property name="defaultTransactionIsolation" value="1"/>
<property name="initialSize" value="0"/>
<property name="maxActive" value="20"/>
<property name="minIdle" value="0"/>
</bean>
Than I have configured:
<bean id="EventDao" class="my.managament.database.class">
<property name="dataSource" ref="dataSource"/>
</bean>
And mainPageController which handles all HTTP requests sent to application
<bean id="mainController" class="my.management.main.controller.class">
In the rest of application, I use gedDatabase() to acquire DB connection, and do select through JDBCTemplate.
Where am I getting wrong?
Thanks
You want to use dao and jdbcTemplate and dataSource through a connection pool. My guess the closest correct approach to your setup is having a dao which has JdbcTemplate field and a JdbcTemplate bean created with your dataSource bean. It would look like:
public class MyDAO {
private JdbcTemplate jdbcTemplate;
// your dao methods using jdbcTemplate here
}
where jdbcTemplate comes from a bean like:
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<constructor-arg ref="dataSource">
</bean>
You should never need to obtain a connection from dataSource (which is apache dbcp based connection pool in your case) directly. JdbcTemplate will get a connection itself when needed. I'm not sure what "gedDatabase" is but it sounds like you tried to get connection yourself and possibly forgot to close it. This would result in pool quickly running out of connections. After handling 20 requests, the subsequent requests would be stuck on attempt to get connection from the pool.
Also I don't understand why and how you see multiple pools. You have a single connection pool which can hold up to 20 connections. All your beans are created as singletons which is the default spring scope.
What is the life time of your EventDao ? You are injecting your DataSource into the dataSource property. I suspect that you are creating multiple EventDao beans and each time you create one you have a new DataSource. I think we would need to have further understanding of the code to be able to answer your question properly.
My two cents:
As far as I am concerned having code and wiring things through XML is a terrible anti-pattern.
Related
I have a Java Spring application connecting to a SQL Server database.
The connection settings are:
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="net.sourceforge.jtds.jdbc.Driver" />
<property name="jdbcUrl"
value="jdbc:jtds:sqlserver://${db.host}:1433/TestDB" />
<property name="user" value="${db.user}" />
<property name="password" value="${db.pass}" />
<!-- these are connection pool properties for C3P0 -->
<property name="minPoolSize" value="10" />
<property name="maxPoolSize" value="100" />
<property name="acquireIncrement" value="5"/>
<property name="maxIdleTime" value="30000" />
</bean>
everything works fine but sometimes I got the following error:
Could not open JDBC Connection for the transaction; nested exception is java.sql.SQLException: I/O Error: Read timed out
I have searched a lot but can't find any clue, any idea or help?
I'm using
<bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
<constructor-arg index="0" ref="sqlSessionFactory" />
</bean>
in my spring-config xml to get my sqlSession, and in the DAO I use:
#Autowired
SqlSession sqlSession;
and then I execute the queries I want. Is it possible that because the connection is not closed that this error exists?
In my case connections were getting dropped when the nightly db backup job triggered. I'm using jtds/sql-server as well. Here is what I did to fix it:
Create/setup a health-check cron job that executes a simple query from within your application, like a short select from. Call it every 10 minutes or so and log the result. It will give you some feedback about when and why this is happening.
Reduce the idle time parameter (maxIdleTime) in your configuration so that old connections get automatically discarded.
Keep in mind that if you don't change the maxIdleTime and you keep multiple connections open, some of them may remain in a bad state even if you are using the health-check function. Quoting from c3p0 documentation:
By default, pools will never expire Connections. If you wish Connections to be expired over time in order to maintain "freshness", set maxIdleTime and/or maxConnectionAge. maxIdleTime defines how many seconds a Connection should be permitted to go unused before being culled from the pool. maxConnectionAge forces the pool to cull any Connections that were acquired from the database more than the set number of seconds in the past.
Another way to setup a health-check call is by using the idleConnectionTestPeriod parameter. Also check this answer which can give you more ideas on how to set it up.
Our application has multiple modules, each module use its own schema in the same mysql database. Now I need to make different connection pool configurations for each module because of their different db resource consuming nature, i.e. some module may have 20 active connections at a point of time, but others may just have 1 max. I have searched here and other forums, couldn't find a solution, just this topic is not about multi-tenancy or multi-database, all schemas are in the same db.
Here's the config we have:
<bean id="dataSource" class="our.own.package.RoutingDataSource"> <!-- RoutingDataSource extends spring AbstractDataSource -->
<property name="master" ref="masterDS"/>
</bean>
<bean id="abstractDataSource" abstract="true">
<property name="driverClass" value="com.mysql.jdbc.Driver" />
<property name="initialPoolSize" value="#initial.pool.size#" />
<property name="minPoolSize" value="#min.pool.size#" />
<property name="maxPoolSize" value="#max.pool.size#" /> <!-- I want to have different configs for each module in our application -->
</bean>
<bean id="masterDS" class="com.mchange.v2.c3p0.ComboPooledDataSource" parent="abstractDataSource">
<property name="jdbcUrl" value="jdbc:mysql://#host#/" />
<property name="user" value="#user#" />
<property name="password" value="#pwd#" />
<property name="dataSourceName" value="#dbName#" />
</bean>
So now I have two questions:
1) Is it possible to have different connection pool configurations for one datasource in Spring?
2) If I have to go with the multiple datasource way(one datasource for one module), is implementing Spring's AbstractRoutingDataSource the correct way to go?
Thank you!
Ad.1 Your data source is in the fact connection pool so you want to have multiple pools on the top of another pool. You can do it but you will face many other problems.
Ad.2. Yes definitely. You already have RoutingDataSource so this one should be implementation of AbstractRoutingDataSource. Probably you already have logic there to determine current the data source routing key which should be used to do the lookup.
I've been researching this a bunch today and I'm starting to think that what I want to do may not be possible, so I am turning to you, o mighty Stackoverflow, for help.
I'm building a RESTful services platform in Java, with Spring Data 3.1.2 + JPA as my persistence layer (as documented here). My data model objects are all implemented as interfaces that extend the Spring JpaRepository interface. I've got everything wired up and working nicely with a single datasource, as shown in this example (note that the datasource shown is Derby, but that's just for development purposes; in production, we'll be using Oracle):
<jpa:repositories base-package="com.my.cool.package.repository"/>
<bean id="emf" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver" />
</property>
<property name="packagesToScan" value="com.my.cool.package" />
<property name="jpaVendorAdapter" ref="jpaVendorAdapter" />
</bean>
<bean name="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="emf" />
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.apache.derby.jdbc.EmbeddedDriver" />
<property name="url" value="jdbc:derby:derbyDB" />
<property name="username" value="dev" />
<property name="password" value="notARealPassword" />
</bean>
<bean id="jpaVendorAdapter" class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="databasePlatform" value="org.hibernate.dialect.DerbyTenSevenDialect" />
</bean>
The problem is that this application will need to connect to several (Oracle) databases. The credentials included with each incoming request will contain a field that tells the application which database to go to in order to fulfill that request. The schemas for each database are the same, so there's no need for separate repository interfaces for each database.
After a fair amount of Googling, it's clear that this is a common scenario. To wit:
multiple databases with Spring Data JPA
Spring + Hibernate + JPA + multiple databases
how to setup spring data jpa with multiple datasources
And here's a blog post by a (former?) Spring developer, which isn't actually relevant to the topic at hand, but someone brings it up in the comments, and the author responds with some info:
http://blog.springsource.org/2011/04/26/advanced-spring-data-jpa-specifications-and-querydsl/#comment-198835
The theme that seems to be emerging is that the way to solve this problem is to define multiple EntityManagerFactories, and to wire each one to the appropriate repositories like so:
<jpa:repositories base-package="com.my.cool.package.repository1" entity-manager-factory-ref="firstEntityManagerFactory" />
<jpa:repositories base-package="com.my.cool.package.repository2" entity-manager-factory-ref="secondEntityManagerFactory" />
However, as I've mentioned, I want to reuse my repository across all of the datasources, so this approach doesn't seem like it would work.
I know that there's no way around having logic in my code that takes the relevant piece of information from the request, and uses it to determine which datasource (or EntityManagerFactory) to use. The part I'm struggling with is how to get a handle to that datasource/EntityManagerFactory and "inject" it into my repository objects. Any ideas?
If you're really using the different DataSourcees in a multi-tenant kind of way (essentially assigning a request to a DataSource and sticking with it for the entire request) you should have a look at AbstractRoutingDataSource. It essentially provides a way to keep a Map of DataSourcees as well as a callback method to return a key to be used to lookup the DataSource to be used eventually. The implementation of this method usually looks up some thread bound key and return that (or even maps that onto a DataSource map key in turn). You just have to make sure some web component binds that key to the thread in the first place.
If you have that in place your Spring configuration just sets up a bean for your sub-class of AbstractRoutingDataSource and pipes the map of DataSources into it. Your Spring Data JPA setup stays the default way. The EntityManagerFactoryBean refers to the AbstractRoutingDataSource and you have a single <jpa:repositories /> element only.
I am attempting to do unit testing of my DAO (using Spring and Hibernate). I am using HSQLDB per this tutorial. The tutorial states that the in-memory HSQLDB database can be initialized using a SQL script but I cannot find information on how to do so in Spring. Here is the pertinent Spring context config:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
<property name="url" value="jdbc:hsqldb:mem:mydb" />
<property name="username" value="sa" />
<property name="password" value="" />
<property name="initialSize" value="5" />
<property name="maxActive" value="10" />
<property name="poolPreparedStatements" value="true" />
<property name="maxOpenPreparedStatements" value="10" />
</bean>
Any help would be appreciated. Thanks.
If you are trying to work with in-memory databases and Spring, there is a new jdbc namespace for Spring 3 that makes working with embedded databases very easy.
The best part is that it acts as a DataSource, so it can easily be dropped in to replace your existing dataSource bean.
<jdbc:embedded-database id="dataSource" type="HSQL">
<jdbc:script location="classpath:schema.sql"/>
<jdbc:script location="classpath:test-data.sql"/>
</jdbc:embedded-database>
If you are more interested in doing this with Java Config, take a look at the EmbeddedDatabaseBuilder (new in Spring 3.0).
#Configuration
public class DatabaseTestConfig {
#Bean
public DataSource dataSource() {
return new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.HSQL)
.addScript("classpath:schema.sql")
.addScript("classpath:test-data.sql")
.build();
}
}
Nicholas answer is perfectly fine, but you can use jdbc namespace to initialize external database as well:
<jee:jndi-lookup id="dataSource" jndi-name="java:comp/env/jdbc/DS"/>
<jdbc:initialize-database data-source="dataSource">
<jdbc:script location="classpath:/META-INF/database/init.sql"/>
</jdbc:initialize-database>
In the tutorial you link to, one of the ways of setting things up is this (after obvious correction):
In-memory from a script: jdbc:hsqldb:file:path-to-file
I think that that would appear to be relevant. I suggest replacing path-to-file with something that looks like a fully-qualified filename…
You could get around this by creating a subclass of BasicDataSource with getters/setters for two new properties, initExecuteSqlFile and destroyExecuteSqlFile, that can have a comma-seperated list of SQL files to execute. The subclass would have init() and destroy() methods that handle the init/destroy SQL files.
Then use the following bean definition:
<bean
id="datasource"
class="com.example.MyBasicDataSource"
destroy-method="destroy"
init-method="init"
>
<property name="destroyExecuteSqlFile">
<value>h2-destroy-01.sql</value>
</property>
<property name="initExecuteSqlFile">
<value>h2-init-01.sql,h2-init-02.sql,h2-init-03.sql</value>
</property>
<!-- Other properties -->
</bean>
With embedded-database we would only be able to connect to the database from the same JVM. If we have two JVMs, for performance or other constraints, we can:
Instead of using an embedded-database, you can use the datasource suggested in this answer.
Then initialize like Poitrek De suggested (and suggested in previous answer too). You may want to create tables only if they do not exist (as suggested here).
I develop an applivation with very load(request).
I used following technologies in my appliation:
Jpa/Hibernate as persistense layer
Spring and Spring Dao
C3p0 as connection pooling
my problem is : I run my application , when number of request increase, throw exception in
persistense layer that"Cannt open connection"
I increase oracle max session but my problem not solve
I indept in C3p0 document and test its options but my problem not solve.
Thank you for your attention
You increased max sessions on Oracle, but you didn't increase the max size of your connection pool. The exception is telling you that your pool is exhausted. Either find what's holding connections open and get them released sooner, or increase the number of max active connections in the pool.
Is it possible for you to post the Spring configuration for your DataSource. I would expect something like:
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="oracle.jdbc.driver.OracleDriver"/>
<property name="jdbcUrl" value="${jdbc.connection.url}"/>
<property name="user" value="${jdbc.connection.username}"/>
<property name="password" value="${jdbc.connection.password}"/>
<property name="initialPoolSize" value="5"/>
<property name="minPoolSize" value="5"/>
<property name="maxPoolSize" value="100"/>
</bean>
With another bean configured where the dataSource is passed by reference:
<bean id="mySampleDao" class="com.example.dao.MySampleDao">
<property name="dataSource" ref="dataSource" />
</bean>
Is this what you have?
What version of Oracle are you using?