I am currently using spring-batch, and there has been an error on our test servers for several month. We don't have direct access to these server, and I can't reproduce this error in our environment, all I have is this stacktrace (see after).
My main problem is, this exception stops the whole batch. Is there a work-around ? some way to catch it and continue ?
The problem does not directly come from one of our reader/processer/writer, it's between them, and I don't know where, I'm guessing during the write, but there is this "at java.net.SocketInputStream.read(SocketInputStream.java:168)" ...
Here is the full stacktrace (more informations after this):
14/04/2016 15:18:46.322 [ERROR] myApp - org.springframework.batch.core.step.AbstractStep Encountered an error executing step myStep1 in job myJob
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:271) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:77) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:368) ~[spring-batch-infrastructure-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144) ~[spring-batch-infrastructure-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:198) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:165) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:362) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:590) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:170) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
... 21 common frames omitted
Caused by: org.hibernate.exception.JDBCConnectionException: Could not open connection
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:235) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1435) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
... 23 common frames omitted
Caused by: org.postgresql.util.PSQLException: La tentative de connexion a échoué. (Note: "The connection attempt failed" in english)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:225) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.Driver.makeConnection(Driver.java:393) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.Driver.connect(Driver.java:267) ~[postgresql-9.2-1002.jdbc4.jar:na]
at java.sql.DriverManager.getConnection(DriverManager.java:582) ~[na:1.6.0_45]
at java.sql.DriverManager.getConnection(DriverManager.java:154) ~[na:1.6.0_45]
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:153) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:144) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:155) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:120) ~[spring-jdbc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
... 28 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[na:1.6.0_45]
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:376) ~[postgresql-9.2-1002.jdbc4.jar:na]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) ~[postgresql-9.2-1002.jdbc4.jar:na]
... 45 common frames omitted
As notes and side questions:
When i first wrote the post, the question was "I want to know when/where in the job this exception is thrown ?". I could use this to prevent the crash, but actually, any solution is good.
Other datas: The stacktrace is the main clue I have. The spring batch tables indicates that 79000 items have been read (pagesize is 50000 for the reader but commit-interval is 500) until it crashes.
So I don't know if the tables are up to date when evrything stops.
Yes, I'm using DriverManagerDataSource which is not a viable pool manager. I'm not sure a real connection pool manager would change anything (like, retry on fail or whatever ? does that exist ? would it work here ?), and most informations on DBCP and c3p0 seems outdated (any recommandation if one could prevent the error would be great). 99.9% of the data throws an exception in the processor for business/flow control (I know, but can't change) and there are some skippable-exception-classes, but they are custom exceptions.
For the exception that causes the whole thing to crash (org.postgresql.util.PSQLException and java.net.SocketException: Connection reset) I can't reproduce it, neither see why the connection is reset: I've seen some thread about this error, but I can't access the postgresql logs, and some people said that this error might even be due to a proxy that sets a timeout on its own (or something like that). I've even looked at the source code of the classes from the last stack trace displayed, but it didn't help. So I don't think I can do anything about this error.
Thank you.
JDBCConnectionException: Connection refused or Connection timed out which means Database is either down or rejecting connection.
First of all check the data sources is being used in your applications, including in all config files is used in hibernate xml's. There might be possibility the database may not be reachable. Please check the IP address or host name, Port, DB may not accept TCP/IP connections, firewall or proxy.
Related
Environment
Database: Cloud SQL(Postgres) on GCP
Application: ServiceA (written in java 8)
Below is the simplified piece of code where the error is happening:
for(String id: ids){
int result = dao.getCountForId(id) // where the error is thrown
if(result>0){
...
}
}
The query issued to Postgres:
SELECT COUNT(*) FROM t WHERE id = ?
The error throw is like below:
xxxxx.postgres.PostgresRuntimeException: Failed get count for id 7dddfdsfdsewr345234sd.........
xxxx.postgres.PostgresRuntimeException: Failed get result for id ....
Caused by: org.PostgreSQL.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:339) ~[app.jar:?]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448) ~[app.jar:?]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369) ~[app.jar:?]
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:153) ~[app.jar:?]
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:103) ~[app.jar:?]
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.postgresql.ds.PGPooledConnection$StatementHandler.invoke(PGPooledConnection.java:428) ~[app.jar:?]
at com.sun.proxy.$Proxy40.executeQuery(Unknown Source) ~[?:?]
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:372) ~[app.jar:?]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2043) ~[app.jar:?]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312) ~[app.jar:?]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448) ~[app.jar:?]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369) ~[app.jar:?]
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:153) ~[app.jar:?]
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:103) ~[app.jar:?]
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.postgresql.ds.PGPooledConnection$StatementHandler.invoke(PGPooledConnection.java:428) ~[app.jar:?]
This error happen in production when the number of ids was high. But I am not able to replicate in a test or local environment.
Attempts
I thought the cause of the error was a lot of queries issued due to large array ids that caused the connection pools of Postgres to saturate. To validate this hypothesis I created an integration test that saturates the connection pool with a lot of queries like below(simplified version):
//FILE_PATH contains the input and expected output for the test
#Test
#FileParameters(value = FILE_PATH)
public test(String taskId, int expected){
int result = dao.getCountForId(id)
Assert.assertEquals(expected,result);
}
The error from the saturated connection pool with a lot of queries is like below:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:525) ~[postgresql-42.2.14.jar:42.2.14]
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146) ~[postgresql-42.2.14.jar:42.2.14]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197) ~[postgresql-42.2.14.jar:42.2.14]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.14.jar:42.2.14]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217) ~[postgresql-42.2.14.jar:42.2.14]
at org.postgresql.Driver.makeConnection(Driver.java:458) ~[postgresql-42.2.14.jar:42.2.14]
at org.postgresql.Driver.connect(Driver.java:260) ~[postgresql-42.2.14.jar:42.2.14]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_282]
at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[?:1.8.0_282]
at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:98) ~[postgresql-42.2.14.jar:42.2.14]
at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:83) ~[postgresql-42.2.14.jar:42.2.14]
I search on the logs of GCP for the error org.PostgreSQL.util.PSQLException: FATAL: sorry, too many clients already but didn't found any records. It seems that this error is not thrown.
I will appreciate any help or guidance in solving this issue.
Best Regards,
Rando.
In my project I wish to have api end point to get order detail.Internally it use ORM and database connection pooling. The process works but some time once in ten days I get following error message and api processing failed.Any idea what could be the problem and possible solution?
Tool used
Apache Karaf (4.1.2)
Apache cxf for api
hibernate 5.2.10
Additional bundles
167 │ Active │ 80 │ 2.1.0 │ Apache Aries Transaction Blueprint
168 │ Active │ 80 │ 1.3.3 │ Apache Aries Transaction Manager
244 │ Active │ 80 │ 1.1.0 │ OPS4J Pax JDBC Pooling Aries Transaction JDBC
Connection pooling (OPS4J Pax Jdbc)
osgi.jdbc.driver.name=mariadb
dataSourceName=Order
databaseName=order_database
user=user1
password=user1
pool=aries
xa=true
url=jdbc:mysql://ip/mydb?characterEncoding=UTF-8&autoReconnect=true&autoReconnectForPools=true
Persistence.xml
<persistence-unit name="myorder" transaction-type="JTA">
<provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
<jta-data-source>osgi:service/javax.sql.DataSource/(osgi.jndi.service.name=myorder)</jta-data-source>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQLInnoDBDialect"/>
</properties>
</persistence-unit>
Error log stack trace
| ERROR | qtp1522571225-12593 | OrderServiceRest | 126 - com.mycompany.olm.Order-application - 1.0.0.0 | Get Order Api failed for orderNo=1101
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not prepare statement
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:147) [230:org.hibernate.core:5.2.10.Final]
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:155) [230:org.hibernate.core:5.2.10.Final]
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1423) [230:org.hibernate.core:5.2.10.Final]
at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1463) [230:org.hibernate.core:5.2.10.Final]
at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getSingleResult(CriteriaQueryTypeQueryAdapter.java:107) [230:org.hibernate.core:5.2.10.Final]
at com.mycompany.olm.order.repository.GenericRepository.findUniqueExistingEntityByPropertyValueEquality(GenericRepository.java:109) [130:com.mycompany.olm.Order-infrastructure:1.0.0.0]
at sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]
at org.apache.aries.proxy.impl.ProxyHandler$1.invoke(ProxyHandler.java:54) [36:org.apache.aries.proxy:1.1.1]
at org.apache.aries.proxy.impl.ProxyHandler.invoke(ProxyHandler.java:119) [36:org.apache.aries.proxy:1.1.1]
at com.mycompany.olm.order.repository.$OrderRepo1496888859.findUniqueExistingEntityByPropertyValueEquality(Unknown Source) [130:com.mycompany.olm.Order-infrastructure:1.0.0.0]
at com.mycompany.olm.persistence.OrderPersistenceServiceImpl.findByOrderNumber(OrderPersistenceServiceImpl.java:23) [130:com.mycompany.olm.Order-infrastructure:1.0.0.0]
at Proxy4e2856b2_5257_441c_8dcd_b768d177afda.findByOrderNumber(Unknown Source) [?:?]
at com.mycompany.olm.order.service.impl.OrderServiceImpl.findByOrderNumber(OrderServiceImpl.java:27) [131:com.mycompany.olm.Order-service:1.0.0.0]
at Proxy3f8ccbf8_c3f9_4d67_9aac_c3e79875e323.findByOrderNumber(Unknown Source) [?:?]
at Proxyf1210a3a_710e_42c7_af23_7e762904107c.findByOrderNumber(Unknown Source) [?:?]
at com.mycompany.olm.order.application.boundary.OrderServiceRest.getOrder(OrderServiceRest.java:61) [126:com.mycompany.olm.Order-application:1.0.0.0]
at sun.reflect.GeneratedMethodAccessor377.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) [51:org.apache.cxf.cxf-core:3.1.11]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [51:org.apache.cxf.cxf-core:3.1.11]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189) [52:org.apache.cxf.cxf-rt-frontend-jaxrs:3.1.11]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99) [52:org.apache.cxf.cxf-rt-frontend-jaxrs:3.1.11]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) [51:org.apache.cxf.cxf-core:3.1.11]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) [51:org.apache.cxf.cxf-core:3.1.11]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) [51:org.apache.cxf.cxf-core:3.1.11]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [51:org.apache.cxf.cxf-core:3.1.11]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:262) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:223) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [30:javax.servlet-api:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274) [59:org.apache.cxf.cxf-rt-transports-http:3.1.11]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [92:org.eclipse.jetty.servlet:9.3.14.v20161028]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [92:org.eclipse.jetty.servlet:9.3.14.v20161028]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [115:org.ops4j.pax.web.pax-web-jetty:6.0.6]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [89:org.eclipse.jetty.security:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [115:org.ops4j.pax.web.pax-web-jetty:6.0.6]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [92:org.eclipse.jetty.servlet:9.3.14.v20161028]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [115:org.ops4j.pax.web.pax-web-jetty:6.0.6]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.server.Server.handle(Server.java:534) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [91:org.eclipse.jetty.server:9.3.14.v20161028]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [84:org.eclipse.jetty.io:9.3.14.v20161028]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [84:org.eclipse.jetty.io:9.3.14.v20161028]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [84:org.eclipse.jetty.io:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [94:org.eclipse.jetty.util:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [94:org.eclipse.jetty.util:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [94:org.eclipse.jetty.util:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [94:org.eclipse.jetty.util:9.3.14.v20161028]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [94:org.eclipse.jetty.util:9.3.14.v20161028]
at java.lang.Thread.run(Thread.java:745) [?:?]
Caused by: org.hibernate.exception.GenericJDBCException: could not prepare statement
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1940) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1909) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doQuery(Loader.java:932) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doList(Loader.java:2615) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doList(Loader.java:2598) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.list(Loader.java:2425) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:502) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:379) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:216) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1488) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1445) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1414) ~[230:org.hibernate.core:5.2.10.Final]
... 61 more
Caused by: java.sql.SQLException: createStatement() is called on closed connection
at org.mariadb.jdbc.MariaDbConnection.checkConnection(MariaDbConnection.java:240) ~[239:org.mariadb.jdbc:1.5.9.0]
at org.mariadb.jdbc.MariaDbConnection.internalPrepareStatement(MariaDbConnection.java:430) ~[239:org.mariadb.jdbc:1.5.9.0]
at org.mariadb.jdbc.MariaDbConnection.prepareStatement(MariaDbConnection.java:271) ~[239:org.mariadb.jdbc:1.5.9.0]
at org.tranql.connector.jdbc.ConnectionHandle.prepareStatement(ConnectionHandle.java:241) ~[?:?]
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1940) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1909) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doQuery(Loader.java:932) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doList(Loader.java:2615) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.doList(Loader.java:2598) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.Loader.list(Loader.java:2425) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:502) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:379) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:216) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1488) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1445) ~[230:org.hibernate.core:5.2.10.Final]
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1414) ~[230:org.hibernate.core:5.2.10.Final]
... 61 more
The process works but some time once in ten days I get following error message and api processing failed.Any idea what could be the problem and possible solution?
Many database connections have a maximum lifetime. Once they have been open for that long they are automatically closed. MariaDB (the one that you reference) also closes connections that have been idle for a period of time. If your pool has not been configured to evict connections that are too old, or to evict idle connections more often than the back-end would close them, then you can end up in the situation where dead connections are in the pool. These would normally get picked up if your pool tests the connections before returning them, but you may not have a test query registered either.
In summary, you can probably configure around this by setting a maximum connection lifetime on the pool and/or an idle timeout.
These details are actually covered in the latest OSGi R7 standard called Transaction Control. This provides a robust, managed lifecycle for resources (including JPA) with a more reliable integration model for modular Transaction Management. You may wish to look at this, or simply at the documentation for your pooling provider.
I'm using Spring and Hibernate with MySQL and I have a problem with its timeout.
I used autoReconnect=true at the end of database URL but I still receive the exception like this below:
2016-08-03 01:01:00 - SCHEDULED ACTIVITY TO DELETE OLD NOTIFICATIONS
2016-08-03 01:01:00 - Threw exception in WakeUpDatabase::smartQuery :com.exception.QueryException: Error using database
at com.services.dbservices.DatabaseNotificationServicesImpl.findAllNotifications(DatabaseNotificationServicesImpl.java:160)
at com.services.NotificationManagementServicesImpl.deleteOldNotifications(NotificationManagementServicesImpl.java:47)
at com.services.WakeUpDatabase.smartQuery(WakeUpDatabase.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.TransactionException: JDBC begin transaction failed:
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodIntercceptor.invoke(CrudMethodMetadataPostProcessor.java:122)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy286.findAll(Unknown Source)
at com.services.dbservices.tables.NotificationServicesImpl.findAllNotifications(NotificationServicesImpl.java:37)
at com.services.dbservices.DatabaseNotificationServicesImpl.findAllNotifications(DatabaseNotificationServicesImpl.java:158)
... 16 more
Caused by: javax.persistence.PersistenceException: org.hibernate.TransactionException: JDBC begin transaction failed:
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771)
at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64)
at org.springframework.orm.jpa.DefaultJpaDialect.beginTransaction(DefaultJpaDialect.java:67)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
... 31 more
Caused by: org.hibernate.TransactionException: JDBC begin transaction failed:
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:76)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1471)
at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61)
... 33 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 46,107,953 milliseconds ago. The last packet sent successfully to the server was 46,107,953 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:400)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1038)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3621)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2429)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2594)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2541)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4882)
at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:371)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAutoCommit(PoolingDataSource.java:328)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:72)
... 36 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3603)
... 43 more
This is the database bean configuration:
public class AppConfig extends WebMvcConfigurerAdapter implements AsyncConfigurer
....
#Bean(name = "dataSource")
public BasicDataSource dataSource() {
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName(env.getRequiredProperty(PROPERTY_NAME_DATABASE_DRIVER));
ds.setUrl(env.getRequiredProperty(PROPERTY_NAME_DATABASE_URL));
ds.setUsername(env.getRequiredProperty(PROPERTY_NAME_DATABASE_USERNAME));
ds.setPassword(env.getRequiredProperty(PROPERTY_NAME_DATABASE_PASSWORD));
return ds;
}
In my application.properties I have
#DB properties:
db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://dbAddress:3306/schemaName?autoReconnect=true
db.username=username
db.password=password
How can I avoid this tedious exception?Thanks
UPDATE: I add
ds.setValidationQuery("SELECT 1");
to my configuration, now I have to see if the exception will be thrown again
You see the following link about the exception :-Why does Hibernate/JDBC/MySQL drop connections after a day or so?
Try to using the setting following property in hibernate configuration
property name="connection.autoReconnect" = true
property name="connection.autoReconnectForPools"= true
property name="connection.is-connection-validation-required" = true
If do not want to use the autoReconnect and wait_time , then go for the ,its is only the suggestion, schedular.
Make the scheduler which run after every 7 hours and run some test query.This make the MySql server active forever.
I have a web application that is being tested for a while on customer's servers. Everything works fine until something triggers a sticky exception that raises only in certain situations (exact ui page, each time random). From it's stacktrace, exception is caused by broken connection to a database (ORA-12519, TNS:no appropriate service handler found), but I can't figure out why - other pages work fine with same db query patterns.
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:245) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) ~[spring-tx-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:40) ~[camunda-engine-spring-7.2.0.jar:7.2.0]
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) ~[camunda-engine-7.2.0.jar:7.2.0]
at org.camunda.bpm.engine.impl.RuntimeServiceImpl.getVariablesTyped(RuntimeServiceImpl.java:172) ~[camunda-engine-7.2.0.jar:7.2.0]
at org.camunda.bpm.engine.impl.RuntimeServiceImpl.getVariables(RuntimeServiceImpl.java:168) ~[camunda-engine-7.2.0.jar:7.2.0]
at org.camunda.bpm.engine.impl.RuntimeServiceImpl.getVariables(RuntimeServiceImpl.java:60) ~[camunda-engine-7.2.0.jar:7.2.0]
at kz.primesource.platform.blocks.util.function.BpmUtil.processVariables(BpmUtil.java:41) ~[platform-blocks-bpm-camunda-1.0-SNAPSHOT.jar:na]
at sun.reflect.GeneratedMethodAccessor815.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
at de.odysseus.el.tree.impl.ast.AstFunction.invoke(AstFunction.java:113) ~[juel-impl-2.2.7.jar:2.2.7]
at de.odysseus.el.tree.impl.ast.AstFunction.eval(AstFunction.java:120) ~[juel-impl-2.2.7.jar:2.2.7]
... 83 common frames omitted
Caused by: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:489) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:553) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:254) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:528) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_45]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[na:1.8.0_45]
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:153) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:144) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:155) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:120) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:204) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
... 96 common frames omitted
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:399) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1140) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:340) ~[ojdbc6-11.2.0.4.0.jar:11.2.0.4.0]
... 107 common frames omitted
I use server Tomcat + HttpServlet + hibernate. I test the server performance, for this, I created an application that simulates the operation 500 clients, each client with a periodicity repeats HTTP requests to the server.
when requests connection = 900 (+-100) i start getting hibernate errors
24.04.2013 12:27:20 org.hibernate.engine.jdbc.spi.SqlExceptionHelper logExceptions
WARN: SQL Error: 0, SQLState: 08001
24.04.2013 12:27:23 org.hibernate.engine.jdbc.spi.SqlExceptionHelper logExceptions
org.hibernate.exception.JDBCConnectionException: Could not open connection
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:131)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:304)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.extractPhysicalConnection(ConnectionProxyHandler.java:82)
at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.continueInvocation(ConnectionProxyHandler.java:138)
at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
at $Proxy24.prepareStatement(Unknown Source)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:147)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:166)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:145)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1711)
at org.hibernate.loader.Loader.doQuery(Loader.java:828)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:289)
at org.hibernate.loader.Loader.doList(Loader.java:2438)
at org.hibernate.loader.Loader.doList(Loader.java:2424)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2254)
at org.hibernate.loader.Loader.list(Loader.java:2249)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:470)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:355)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1248)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
at org.hibernate.internal.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:890)
at actions.player.LoginAction.perform(LoginAction.java:20)
at action.Action.performAction(Action.java:66)
at servlets.RequestHandlerServlet.service(RequestHandlerServlet.java:34)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:999)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.postgresql.util.PSQLException:
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:123)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:28)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:20)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:22)
at org.postgresql.Driver.makeConnection(Driver.java:391)
at org.postgresql.Driver.connect(Driver.java:265)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at org.hibernate.service.jdbc.connections.internal.DriverManagerConnectionProviderImpl.getConnection(DriverManagerConnectionProviderImpl.java:193)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:281)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
... 41 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at org.postgresql.core.PGStream.<init>(PGStream.java:60)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:74)
... 54 more
org.hibernate.exception.JDBCConnectionException: Could not open connection
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:131)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:304)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.extractPhysicalConnection(ConnectionProxyHandler.java:82)
please help, any ideas ...
The specific problem is:
Caused by: java.net.ConnectException: Connection refused: connect
This shows that when PgJDBC attempted to open a TCP/IP socket to the PostgreSQL server the connection was rejected.
This doesn't happen when the PostgreSQL server is overloaded; it'll instead accept the connection for long enough to reply with an error mentioning "too many connections" or "max_connections". When I tested with psql against Pg 9.3-HEAD I got:
FATAL: sorry, too many clients already
Similarly, if authentication is a problem, PostgreSQL will accept the connection, reply with an error message, and close the connection, like:
FATAL: no pg_hba.conf entry for host "[local]", user "craig", database "postgres"
or:
FATAL: password authentication failed for user "craig"
What you are seeing is a lower level issue in your OS or networking stack. It's happening before PostgreSQL gets a chance to accept the connection. Since you have not mentioned your operating system, whether the Pg server is local or remote to the app server, or pretty much anything else it's hard to help further with that. I'd start by:
Checking the PostgreSQL logs for errors about accept() or anything else unusual-looking
Investigating any TCP/IP connection limits and max open file limits in the operating system
Making sure I wasn't using any kind of dodgy 3rd party software firewall or "Internet security" suite
Furthermore, you should investigate using a connection pooler - either an in-app one, or an external pooler like PgPool-II, allowing you to reduce max_connections to an efficient level. PostgreSQL will perform better with fewer connections; the ideal number depends on workload and hardware. See PostgreSQL wiki: Number of database connections.