I am facing a problem while using the DefaultMessageListenerContainer in Spring JMS with TIBCO queues. When I put a relatively small message on a queue, the listener on the queue takes a few seconds to read the message and complete the procesing and everthing works smoothly. On the other hand, when I put a request on the queue that may take longer to be processed by the business layer, I get the following exception a few minutes after the processing begins.
04:38:14.392 [TIBCO EMS TCPLink Reader (Server-34171)] WARN o.s.j.c.CachingConnectionFactory - Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: Connection has been terminated
at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:509) ~[tibjms.jar:6.3.0]
at com.tibco.tibjms.TibjmsConnection._invokeOnExceptionCallback(TibjmsConnection.java:2025) [tibjms.jar:6.3.0]
at com.tibco.tibjms.TibjmsConnection._onDisconnected(TibjmsConnection.java:2394) [tibjms.jar:6.3.0]
at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventDisconnected(TibjmsConnection.java:349) [tibjms.jar:6.3.0]
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:330) [tibjms.jar:6.3.0]
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) [tibjms.jar:6.3.0]
The processing continues on the listener side and I get the following exception once the listener has finished processing the message :
07:28:04.281 [jmsContainer-1] WARN o.s.j.l.DefaultMessageListenerContainer - Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.IllegalStateException: Session is closed
at com.tibco.tibjms.TibjmsxSessionImp.getTransacted(TibjmsxSessionImp.java:4837) ~[tibjms.jar:6.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_03]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_03]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_03]
at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_03]
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at $Proxy20.getTransacted(Unknown Source) ~[na:na]
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:572) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:481) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) [org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1059) [org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1051) [org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:948) [org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_03]
07:28:04.281 [jmsContainer-1] WARN o.s.j.l.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'Queue[cmb.cmbtech.na.bead_153036.rwacalrequest1]' - trying to recover. Cause: Session is closed
javax.jms.IllegalStateException: Session is closed
at com.tibco.tibjms.TibjmsxSessionImp.getTransacted(TibjmsxSessionImp.java:4837) ~[tibjms.jar:6.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_03]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_03]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_03]
at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_03]
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at $Proxy20.getTransacted(Unknown Source) ~[na:na]
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:572) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:481) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1059) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1051) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:948) ~[org.springframework.jms-3.1.0.RELEASE.jar:3.1.0.RELEASE]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_03]
And finally, a few seconds after the above exception, I get the following info in the logs
07:28:04.418 [jmsContainer-1] INFO o.s.j.l.DefaultMessageListenerContainer - Successfully refreshed JMS Connection
I am not really sure what is causing the probelm but it seems that the jms connection is getting reset whenever the listener takes more than a few minutes to process the message. The side effect of all this is that the listener fails to publish the processing output to another queue for an external application to consume. Do I need to set a transaction manager for my listener or set some kind of property that will not reset the connection for a given duration?
Related
What is the recommended way to make Spring Kafka reload SSL context?
I have a requirement to insert new certificates into the trust store that my Kafka producer uses without any downtime.
However what I have found is that once the application is started and a Kafka producer is created, an instance of SSLContext is created and cached. There is a way to reconfigure this but the only way I have found so far is to destroy any existing producers by invoking the destroy method on DefaultKafkaProducerFactory (after certificate renewal) which causes any subsequent calls to KafkaTemplate.send to force a new producer to be created which in turn reloads the SSL context.
I feel this is like using a sledgehammer to solve this problem and there might be a more elegant solution. I have also noticed that if I call destroy when there are messages being sent, I get the below exception which doesn't look very positive when we cannot afford to lose any events.
java.util.concurrent.CompletionException: org.apache.kafka.common.KafkaException: Producer closed while send in progress
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1592)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: org.apache.kafka.common.KafkaException: Producer closed while send in progress
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:826)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:444)
at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:372)
at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:190)
at org.springframework.kafka.core.KafkaOperations$send.call(Unknown Source)
at com.example.event.publisher.kafka.KafkaEventPublisher.doPublish(KafkaEventPublisher.groovy:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:352)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:68)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:177)
at com.example.event.publisher.kafka.KafkaEventPublisher$_publish_closure1.doCall(KafkaEventPublisher.groovy:47)
at com.example.event.publisher.kafka.KafkaEventPublisher$_publish_closure1.doCall(KafkaEventPublisher.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at groovy.lang.Closure.call(Closure.java:418)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
at com.sun.proxy.$Proxy103.get(Unknown Source)
at java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
... 6 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: Requested metadata update after close
at org.apache.kafka.clients.Metadata.awaitUpdate(Metadata.java:200)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:938)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:823)
... 37 common frames omitted
It looks like simply resetting the config for certain values will trigger a rebuild on the SSL engine factory. They even call out file key config that would cause a hot reload.
link
Spring Kafka 2.6.5, Kafka 2.4.1, I use KafkaProducerFactory.reset() instead.
#Autowired
private final KafkaTemplate<String, byte[]> kafkaTemplate;
private void reloadProducer() {
kafkaTemplate.getProducerFactory().reset();
}
Next time send() is called, Spring will recreate a brand spanking new KafkaConsumer with the new certificate.
I'm using FtpStreamingMessageSource in combination with poller with following config (#InboundChannelAdapter(channel = "ftpChannel", poller = #Poller("pollerMetadata"))):
#Bean
public PollerMetadata pollerMetadata(PlatformTransactionManager transactionManager) {
PeriodicTrigger trigger = new PeriodicTrigger(TimeUnit.SECONDS.toMillis(30));
trigger.setFixedRate(true);
MatchAlwaysTransactionAttributeSource source = new MatchAlwaysTransactionAttributeSource();
source.setTransactionAttribute(new DefaultTransactionAttribute());
TransactionInterceptor interceptor = new TransactionInterceptor(transactionManager, source);
PollerMetadata metadata = new PollerMetadata();
metadata.setTrigger(trigger);
metadata.setTransactionSynchronizationFactory(synchronizationFactory());
metadata.setAdviceChain(Collections.singletonList(interceptor));
return metadata;
}
It was working OK, until today I had a DB problem and an exception The last packet sent successfully to the server was 30,079 milliseconds ago. and (ERROR): LoggingHandler org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Connection is
closed.
Somehow, poller stopped working after this, even though HikariCP managed to recover from exception after a while. Seems like a thread that was doing polling job was terminated.
Any idea how to make thread recover and continue with processing? After I restarted application, everything was back to normal.
UPDATE
this is the last exception I got before poller stopped working
2019-03-06 14:34:45 (ERROR): LoggingHandler org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Connection is
closed
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:290)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:853)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:830)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:503)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:285)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy71.call(Unknown Source)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:51)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:344)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy67.rollback(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:370)
at com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:287)
... 22 more
There is no "termination" of a polling thread; polling uses a TaskScheduler and when the poll completes (whether an exception occurred or not) the thread is returned to the pool, ready for the next poll.
It's probably too late now (if you restarted your app) but if it happens again take a thread dump; most likely the poller thread is "stuck" somewhere in user (or DB) code.
I'm the new to redis, I start the server about this tutorial. And it work. Then I use write the code using java to connect redis, then it's ok, like this:
Jedis jedis = new Jedis("localhost");
System.out.println("Connection to server sucessfully");
//store data in redis list
jedis.lpush("tutorial-list", "Redis");
jedis.lpush("tutorial-list", "Mongodb");
jedis.lpush("tutorial-list", "Mysql");
But, when I use multithread to push the redis, it will throw the exception "read time out":
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601) at
org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException:
java.net.SocketTimeoutException: Read timed out at
redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:201)
at
redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
at redis.clients.jedis.Protocol.process(Protocol.java:141) at
redis.clients.jedis.Protocol.read(Protocol.java:205) at
redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
at
redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:233)
at redis.clients.jedis.Jedis.keys(Jedis.java:185) at
org.v11.redis_mongo_task.UpdateApp.jobDetail(UpdateApp.java:23) at
org.v11.redis_mongo_task.UpdateApp.main(UpdateApp.java:42) ... 5 more
Caused by: java.net.SocketTimeoutException: Read timed out at
java.net.SocketInputStream.socketRead0(Native Method) at
java.net.SocketInputStream.read(SocketInputStream.java:150) at
java.net.SocketInputStream.read(SocketInputStream.java:121) at
java.net.SocketInputStream.read(SocketInputStream.java:107) at
redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:195)
... 13 more
What happened for redis? why it can work in single thread?
According to this answer, a single Jedis instance is not threadsafe. You will have to use JedisPool for multithreading. You can read here on how use it and here to set the max connections and what will happen if those connections are all occupied.
I'm posting links since two of them are SO answers an they should get the credit and one is from github official repo, so if anything gets updated it should be reflected here too.
I'm working on a distributed system which is RMI based using jdk1.6.
Occasionally I can see ConcurrentModificationException errors on that thread happening within the RMI runtime when it fails to serialize objects. I can reproduce that exception easily by concurrently updating object being returned from remote method.
But the problem is that I can't find the source of those calls. RMI exception is written to stderr (captured on server side within runtime code after it exits remote object method), but there are no matching exception in client services (while if that was a legitimate remote call, RemoteException with appropriate cause would be raised).
The only different thing about those exceptions is that they are happening on "RMI TCP Connection(idle)" thread and not on a thread like "RMI TCP Connection(<connection count>)-<client endpoint info>".
Any clues on what are those "idle" threads in RMI? I failed to find such within the openjdk sources.
Upd: I'm adding an exception stack trace as reproduced, which is what you usually see in described situation.
Server side console shows:
Exception dispatching call to [-3534448f:12f54948b7f:-7fff, 349678755005857493] in thread "RMI TCP Connection(6)-x.x.x.x" at Thu Apr 14 16:15:13 BST 2011:
java.util.ConcurrentModificationException
at java.util.ArrayList.writeObject(ArrayList.java:573)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1469)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Client size exception thrown to caller:
java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.io.EOFException
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)
at $Proxy0.getData(Unknown Source)
at Clnt.main(Clnt.java:11)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
at java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:1899)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:155)
... 9 more
Threads are created in RMI's connection pool with the name 'RMI TCP Connection(idle)'. When one of these is used, the runnable it is being used to execute renames the thread to 'RMI TCP Connection(n)' where n is the connection number being handled (a sequential number), and renames back to 'idle' in the runnable's finally block. So any trace labelled 'RMI TCP Connection(idle)' must come before the runnable renames it as a connection thread, or after it has been renamed back.
Don't ask me how that is possible. The actual answer to your problem, if not your question, is not to modify objects while they are concurrently being returned ;-)
I got to the bottom of it. Problem happens in two cases:
When marshaling a return value throws some exception and that exception in turn contains an object prone to concurrent modification. UnicastServerRef tries to write the cause into the (already corrupted) return stream and causes ConcurrentModificationException.
When method raises an exception and this exception fails to serialize with ConcurrentModificationException (or any other runtime exception).
This exception goes all the way up the stack and is caught and logged by thread pool, not RMI runtime (that's why there is no Exception dispatching call to line at the beginning). That explains thread name being idle since it is already returned to pool from RMI's point of view.
Here's the real exception which is in fact a bit different from what reproduced exception shows in terms of first line and actual call trace:
Exception in thread "RMI TCP Connection(idle)" java.util.ConcurrentModificationException
at java.util.ArrayList.writeObject(ArrayList.java:573)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:343)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
It doesn't give much clues to the original problem, so I'll have to wait for a client exception to show up at some point to fix that.
If that kind of exception happens when reading JMX attribute by jconsole, it'll not show stacktrace, but will show attribute value as Unavailable.
I get this error randomly when the web service is accessed.It is difficult to replicate and once I refresh it everything functions as expected.
Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2235)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
at org.hibernate.loader.Loader.list(Loader.java:2124)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1597)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
at edu.asd.myproj.hibernate.adapter.CriteriaAdapter.list(CriteriaAdapter.java:380)
at edu.asd.myproj.DaoHibernateImpl.findByCriteria(DaoHibernateImpl.java:778)
at edu.asd.myproj.DaoHibernateImpl.findByCriteria(DaoHibernateImpl.java:765)
at edu.asd.myproj.DaoHibernateImpl.findByProperty(DaoHibernateImpl.java:361)
at edu.asd.myproj.ProfilesDaoImpl.count(ProfilesDaoImpl.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy27.count(Unknown Source)
at edu.asd.myproj.ResourceProfileLoaderNew.loadProfiles(ResourceProfileLoaderNew.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:297)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:250)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:144)
... 50 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was44323 seconds ago.The last packet sent successfully to the server was 44323 seconds ago, which is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3246)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1812)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2232)
... 79 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
... 90 more
This is my hibernate.properties file
# Turn on for SQL debugging
hibernate.show_sql=true
# Generate schema?
hibernate.hbm2ddl.auto=validate
#============================================================
# C3P0 connection pool configuration
#============================================================
#changed to default values
c3p0.acquireIncrement=hibernate.c3p0.acquire_increment
c3p0.idleConnectionTestPeriod=hibernate.c3p0.idle_test_period
c3p0.initialPoolSize=10
c3p0.maxIdleTime=hibernate.c3p0.timeout
c3p0.maxPoolSize=hibernate.c3p0.max_size
c3p0.maxStatements=hibernate.c3p0.max_statements
c3p0.minPoolSize=hibernate.c3p0.min_size
c3p0.testConnectionsOnCheckout=hibernate.c3p0.validate
The error is seen randomly once and it takes a long time before it reappears.It goes away once refreshed.
Please send complete exception stack trace for more understanding of reader.
This exception occured usually when a DB Connection stale.
Stale means: connection killed by the server, but still considered alive by the pool. You need to configure connection testing in the connection. For this approach in C3P0 you have to using following configuration:
c3p0.testConnectionOnCheckin = true
testConnectionOnCheckout= true
and for more checking increase checkout timeout as following:
c3p0.checkoutTimeout = 0
and test another test options in C3P0 such as :
connectionTesterClassName
idleConnectionTestPeriod
automaticTestTable
preferredTestQuery
Another way for test is writing a sample code and connect to database by a JDBC pure codes.(by DriverManager and Connection and etc )