I am using MongoDB to store data. In my project I am connecting multiple users(100-2000 users) to a server using thread pool and store the responses of those users in the database. But when I do this I am getting this error. Please help me with this. I am not able to get it. I have already wasted 45 hours on this error.
Error:
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.avaya.onex.hss.requesthandlers.CommandExecutor.executeCommands(CommandExecutor.java:129)
at com.avaya.onex.hss.requesthandlers.CommandExecutor.run(CommandExecutor.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.mongodb.DBPortPool$SemaphoresOut: Out of semaphores to get db connection
at com.mongodb.DBPortPool.get(DBPortPool.java:176)
at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:370)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:149)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:138)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:261)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:211)
at com.mongodb.DBCollection.insert(DBCollection.java:57)
at com.mongodb.DBCollection.insert(DBCollection.java:102)
at com.avaya.onex.hss.objects.LoginRequest.storeData(LoginRequest.java:152)
at com.avaya.onex.hss.requesthandlers.LoginHandler.handleRequest(LoginHandler.java:20)
... 8 more
com.mongodb.DBPortPool$SemaphoresOut: Out of semaphores to get db connection
at com.mongodb.DBPortPool.get(DBPortPool.java:176)
at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:370)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:212)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:305)
at com.mongodb.DB.command(DB.java:160)
at com.mongodb.DB.command(DB.java:183)
at com.mongodb.DB.command(DB.java:144)
at com.mongodb.DBCollection.drop(DBCollection.java:777)
at com.mongodb.DBApiLayer$MyCollection.drop(DBApiLayer.java:206)
at com.avaya.onex.hss.requesthandlers.CommandExecutor.executeCommands(CommandExecutor.java:118)
at com.avaya.onex.hss.requesthandlers.CommandExecutor.run(CommandExecutor.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Try increasing the number of connections in the connection pool.
Look here, at the connectionsPerHost.
Related
By Hibernate search we create a search engine in my application. Here same application running in two different JBoss and using two different folder for storing index data. Two folder and Two JBoss running in different system. But some time showing bellow type of error. Please, give me any resolution.
16:45:58,184 ERROR
[org.hibernate.search.exception.impl.LogErrorHandler] (Hibernate
Search: Index updates queue processor for index
in.issac.wisebank.systemadmin.customermanagement.entiry.WbSaCustomermaster-1)
HSEARCH000058: Exception occurred
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out: NativeFSLock#/Folder_Path/write.lock Primary Failure: Entity
in.issac.wisebank.systemadmin.customermanagement.entiry.WbSaCustomermaster
Id 49621 Work Type org.hibernate.search.backend.UpdateLuceneWork :
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out:
NativeFSLock#/home/ccblcbs/wisebankenterprise/globalsearch/index12/in.issac.wisebank.systemadmin.customermanagement.entiry.WbSaCustomermaster/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
[lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40] at
org.apache.lucene.index.IndexWriter.(IndexWriter.java:1098)
[lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40] at
org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:146)
[hibernate-search-engine-4.4.4.Final.jar:4.4.4.Final] at
org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:113)
[hibernate-search-engine-4.4.4.Final.jar:4.4.4.Final] at
org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
[hibernate-search-engine-4.4.4.Final.jar:4.4.4.Final] at
org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:101)
[hibernate-search-engine-4.4.4.Final.jar:4.4.4.Final] at
org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
[hibernate-search-engine-4.4.4.Final.jar:4.4.4.Final] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[rt.jar:1.6.0_24] at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[rt.jar:1.6.0_24] at
java.util.concurrent.FutureTask.run(FutureTask.java:138)
[rt.jar:1.6.0_24] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24] at java.lang.Thread.run(Thread.java:662)
[rt.jar:1.6.0_24] 16:45:58,187 ERROR
[org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask]
(Hibernate Search: Index updates queue processor for index
in.issac.wisebank.systemadmin.customermanagement.entiry.WbSaCustomermaster-1)
HSEARCH000072: Couldn't open the IndexWriter because of previous
error: operation skipped, index ouf of sync!
It's caused by Windows, you should read about LOCKING STRATEGY.
https://docs.jboss.org/hibernate/search/3.2/reference/en/html_single/#search-configuration-directory-lockfactories
This version of Hibernate Search is extremely old; this issue could occasionally happen but was resolved. You need to update.
I'm using Hibernate 1.4.191 along with Hibernate 4.3.1. My h2 connection url is dbc:h2:file:./h2/myDB - which is an embedded DB without MULTI_THREADED enabled. I'm running with MVCC on - as this is the default setting for 1.4.191.
Using this configuration in my application, I got the following stack trace while trying to execute an update HQL statement:
2017-07-11 19:38:48 SEVERE Could not set rounding style. org.hibernate.PessimisticLockException: could not execute statement
at org.hibernate.dialect.H2Dialect$2.convert(H2Dialect.java:342)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:190)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:109)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:78)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:445)
at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:347)
at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1282)
at org.hibernate.internal.QueryImpl.executeUpdate(QueryImpl.java:118)
at de.oktopos.dataBase.receipts.converter.ReceiptConverter$29.runOperation(ReceiptConverter.java:1214)
at de.oktopos.dataBase.receipts.converter.ReceiptConverter$29.runOperation(ReceiptConverter.java:1210)
at de.oktopos.dataBase.tools.DatabaseOperationRunner.run(DatabaseOperationRunner.java:71)
at de.oktopos.dataBase.receipts.converter.ReceiptConverter.setRoundingStyle(ReceiptConverter.java:1226)
at de.oktopos.dataBase.receipts.converter.ReceiptCacheWriter.setRoundingStyle(ReceiptCacheWriter.java:99)
at de.oktopos.oktoDeskService.remote.OktoDeskModel.setRoundingStyle(OktoDeskModel.java:841)
at net.oktopos.cashdesk.DeskModelProcessor.setRoundingStyle(DeskModelProcessor.java:1648)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: Zeitüberschreitung beim Versuch die Tabelle zu sperren
Timeout trying to lock table ; SQL statement:
update Receipt set turnoverType=? [50200-191]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.command.Command.filterConcurrentUpdate(Command.java:307)
at org.h2.command.Command.executeUpdate(Command.java:260)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:160)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:146)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:187)
... 29 more
Caused by: org.h2.jdbc.JdbcSQLException: Gleichzeitige Änderung in Tabelle "RECEIPT": eine andere Transaktion hat den gleichen Datensatz geändert oder gelöscht
Concurrent update in table "RECEIPT": another transaction has updated or deleted the same row [90131-191]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.mvstore.db.MVTable.convertException(MVTable.java:898)
at org.h2.mvstore.db.MVSecondaryIndex.remove(MVSecondaryIndex.java:247)
at org.h2.mvstore.db.MVTable.removeRow(MVTable.java:677)
at org.h2.table.Table.updateRows(Table.java:487)
at org.h2.command.dml.Update.update(Update.java:145)
at org.h2.command.CommandContainer.update(CommandContainer.java:98)
at org.h2.command.Command.executeUpdate(Command.java:258)
... 32 more
Caused by: java.lang.IllegalStateException: Entry is locked [1.4.191/101]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773)
at org.h2.mvstore.db.TransactionStore$TransactionMap.set(TransactionStore.java:1031)
at org.h2.mvstore.db.TransactionStore$TransactionMap.remove(TransactionStore.java:989)
at org.h2.mvstore.db.MVSecondaryIndex.remove(MVSecondaryIndex.java:241)
... 37 more
Im trying to understand how this is possible. The stack trace seems to say that a lock could not be acquired because of another connection (50200). In particular it says that another connection concurrently updated the table (90131). But there can only be one connection because the database is embedded. Am I missing something here?
The the HQL query is basically executed as follows (minus error handling boilerplate, DBReceipt maps to receipt):
Session session = sessionFactory.openSession();
Transaction transaction = session.beginTransaction();
session.createQuery("UPDATE DBReceipt SET turnovertype = :turnovertype")
.setParameter("turnovertype", turnovertype)
.executeUpdate();
transaction.commit();
session.close();
There are of course many other database writes/reads in the background on different threads, but all use the same connection.
The error kept happening when the above part of the code was called. Even restarting the application did not help - the next call of the code would cause the same exception. Only dropping the DB helped.
I found the likely source of the problem after looking arround a bit in my code base.
I was namely creating a LockRequest and locking the a receipt in the receipt table. If this lock is never let go (because the transaction ends up in an infinite loop), then any update on the receipt table afterwards would lead to the stack trace above.
I solved the problem by removing the LockRequest (I couldn't find the source of the infinite loop). This was possible because we recently moved to locking that receipt on an application level, and also because there would be no problems in our application with optimistic locking.
Another work around to this problem would be to add a timeout to the lock, transaction, or session. However, my enviroment doesn't seem to support any of these - at least none of them worked in my tests.
I am new to the concept of OLAP DB.
Please excuse me if I missed out something in the Connection String.
I am trying to connect to connect to Microsoft Analysis Server(which is in different Domain) from my System.
While Connecting I am getting below Error.
Can you please let me know, what am I doing wrong?
I have executed the Steps Mentioned in :
https://msdn.microsoft.com/en-us/library/gg492140.aspx
and referred to the below link for connection to the Cube:
http://www.matthewgodding.com/articles/2010/2/27/getting-data-from-sql-analysis-services-using-java-and-olap4.html
After connecting to the CUbe, I need to execute MDX Query.
Please find the snippet of the code I am using:
public static void main(String[] args) {
try {
Class.forName("org.olap4j.driver.xmla.XmlaOlap4jDriver");
final Connection connection = DriverManager
.getConnection("jdbc:xmla:Server=http://Remote_Server_IPAddress/OLAP/msmdpump.dll;"
+ "Catalog=MyCatalog;"
+ "User='Domian\\Username';" \\ This is the Username to login to the Server
+ "Password='Password';" \\ This is the Password to login to the Server
+ "Cube=Cube;"); \\This is the Cube Name
I get the below Exception when I execute the program:
Exception:
java.lang.RuntimeException: org.olap4j.OlapException: This connection encountered an exception while executing a query.
at org.olap4j.driver.xmla.DeferredNamedListImpl.getList(DeferredNamedListImpl.java:96)
at org.olap4j.driver.xmla.DeferredNamedListImpl.size(DeferredNamedListImpl.java:116)
at org.olap4j.driver.xmla.XmlaOlap4jConnection.getOlapDatabase(XmlaOlap4jConnection.java:451)
at org.olap4j.driver.xmla.XmlaOlap4jConnection.getDatabase(XmlaOlap4jConnection.java:444)
at org.sid.practise.CheckConnection.main(CheckConnection.java:79)
Caused by: org.olap4j.OlapException: This connection encountered an exception while executing a query.
at org.olap4j.driver.xmla.XmlaHelper.createException(XmlaHelper.java:43)
at org.olap4j.driver.xmla.XmlaOlap4jConnection.executeMetadataRequest(XmlaOlap4jConnection.java:878)
at org.olap4j.driver.xmla.XmlaOlap4jDatabaseMetaData.getMetadata(XmlaOlap4jDatabaseMetaData.java:137)
at org.olap4j.driver.xmla.XmlaOlap4jDatabaseMetaData.getMetadata(XmlaOlap4jDatabaseMetaData.java:67)
at org.olap4j.driver.xmla.XmlaOlap4jDatabaseMetaData.getDatabaseProperties(XmlaOlap4jDatabaseMetaData.java:1044)
at org.olap4j.driver.xmla.XmlaOlap4jConnection.makeConnectionPropertyList(XmlaOlap4jConnection.java:324)
at org.olap4j.driver.xmla.XmlaOlap4jConnection.generateRequest(XmlaOlap4jConnection.java:1037)
at org.olap4j.driver.xmla.XmlaOlap4jConnection.populateList(XmlaOlap4jConnection.java:849)
at org.olap4j.driver.xmla.DeferredNamedListImpl.populateList(DeferredNamedListImpl.java:136)
at org.olap4j.driver.xmla.DeferredNamedListImpl.getList(DeferredNamedListImpl.java:90)
... 4 more
Caused by: org.olap4j.driver.xmla.proxy.XmlaOlap4jProxyException: This proxy encountered an exception while processing the query.
at org.olap4j.driver.xmla.proxy.XmlaOlap4jHttpProxy.getResponse(XmlaOlap4jHttpProxy.java:181)
at org.olap4j.driver.xmla.proxy.XmlaOlap4jAbstractHttpProxy.get(XmlaOlap4jAbstractHttpProxy.java:181)
at org.olap4j.driver.xmla.proxy.XmlaOlap4jHttpProxy.get(XmlaOlap4jHttpProxy.java:42)
at org.olap4j.driver.xmla.XmlaOlap4jConnection.executeMetadataRequest(XmlaOlap4jConnection.java:876)
... 12 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at org.olap4j.driver.xmla.proxy.XmlaOlap4jHttpProxy.getResponse(XmlaOlap4jHttpProxy.java:136)
It's difficult to know what went wrong for you. Few things you could do,
Don't specify cube in connection. This is not listed as property in XMLA driver.
You can install fiddler and see what's going on with the requests.
Not sure if single quotes are needed in user name and password.
I'm working on a distributed system which is RMI based using jdk1.6.
Occasionally I can see ConcurrentModificationException errors on that thread happening within the RMI runtime when it fails to serialize objects. I can reproduce that exception easily by concurrently updating object being returned from remote method.
But the problem is that I can't find the source of those calls. RMI exception is written to stderr (captured on server side within runtime code after it exits remote object method), but there are no matching exception in client services (while if that was a legitimate remote call, RemoteException with appropriate cause would be raised).
The only different thing about those exceptions is that they are happening on "RMI TCP Connection(idle)" thread and not on a thread like "RMI TCP Connection(<connection count>)-<client endpoint info>".
Any clues on what are those "idle" threads in RMI? I failed to find such within the openjdk sources.
Upd: I'm adding an exception stack trace as reproduced, which is what you usually see in described situation.
Server side console shows:
Exception dispatching call to [-3534448f:12f54948b7f:-7fff, 349678755005857493] in thread "RMI TCP Connection(6)-x.x.x.x" at Thu Apr 14 16:15:13 BST 2011:
java.util.ConcurrentModificationException
at java.util.ArrayList.writeObject(ArrayList.java:573)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1469)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Client size exception thrown to caller:
java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.io.EOFException
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)
at $Proxy0.getData(Unknown Source)
at Clnt.main(Clnt.java:11)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
at java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:1899)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1873)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:155)
... 9 more
Threads are created in RMI's connection pool with the name 'RMI TCP Connection(idle)'. When one of these is used, the runnable it is being used to execute renames the thread to 'RMI TCP Connection(n)' where n is the connection number being handled (a sequential number), and renames back to 'idle' in the runnable's finally block. So any trace labelled 'RMI TCP Connection(idle)' must come before the runnable renames it as a connection thread, or after it has been renamed back.
Don't ask me how that is possible. The actual answer to your problem, if not your question, is not to modify objects while they are concurrently being returned ;-)
I got to the bottom of it. Problem happens in two cases:
When marshaling a return value throws some exception and that exception in turn contains an object prone to concurrent modification. UnicastServerRef tries to write the cause into the (already corrupted) return stream and causes ConcurrentModificationException.
When method raises an exception and this exception fails to serialize with ConcurrentModificationException (or any other runtime exception).
This exception goes all the way up the stack and is caught and logged by thread pool, not RMI runtime (that's why there is no Exception dispatching call to line at the beginning). That explains thread name being idle since it is already returned to pool from RMI's point of view.
Here's the real exception which is in fact a bit different from what reproduced exception shows in terms of first line and actual call trace:
Exception in thread "RMI TCP Connection(idle)" java.util.ConcurrentModificationException
at java.util.ArrayList.writeObject(ArrayList.java:573)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:343)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
It doesn't give much clues to the original problem, so I'll have to wait for a client exception to show up at some point to fix that.
If that kind of exception happens when reading JMX attribute by jconsole, it'll not show stacktrace, but will show attribute value as Unavailable.
I get this error randomly when the web service is accessed.It is difficult to replicate and once I refresh it everything functions as expected.
Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2235)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
at org.hibernate.loader.Loader.list(Loader.java:2124)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1597)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
at edu.asd.myproj.hibernate.adapter.CriteriaAdapter.list(CriteriaAdapter.java:380)
at edu.asd.myproj.DaoHibernateImpl.findByCriteria(DaoHibernateImpl.java:778)
at edu.asd.myproj.DaoHibernateImpl.findByCriteria(DaoHibernateImpl.java:765)
at edu.asd.myproj.DaoHibernateImpl.findByProperty(DaoHibernateImpl.java:361)
at edu.asd.myproj.ProfilesDaoImpl.count(ProfilesDaoImpl.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy27.count(Unknown Source)
at edu.asd.myproj.ResourceProfileLoaderNew.loadProfiles(ResourceProfileLoaderNew.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:297)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:250)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:144)
... 50 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was44323 seconds ago.The last packet sent successfully to the server was 44323 seconds ago, which is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3246)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1812)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2232)
... 79 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
... 90 more
This is my hibernate.properties file
# Turn on for SQL debugging
hibernate.show_sql=true
# Generate schema?
hibernate.hbm2ddl.auto=validate
#============================================================
# C3P0 connection pool configuration
#============================================================
#changed to default values
c3p0.acquireIncrement=hibernate.c3p0.acquire_increment
c3p0.idleConnectionTestPeriod=hibernate.c3p0.idle_test_period
c3p0.initialPoolSize=10
c3p0.maxIdleTime=hibernate.c3p0.timeout
c3p0.maxPoolSize=hibernate.c3p0.max_size
c3p0.maxStatements=hibernate.c3p0.max_statements
c3p0.minPoolSize=hibernate.c3p0.min_size
c3p0.testConnectionsOnCheckout=hibernate.c3p0.validate
The error is seen randomly once and it takes a long time before it reappears.It goes away once refreshed.
Please send complete exception stack trace for more understanding of reader.
This exception occured usually when a DB Connection stale.
Stale means: connection killed by the server, but still considered alive by the pool. You need to configure connection testing in the connection. For this approach in C3P0 you have to using following configuration:
c3p0.testConnectionOnCheckin = true
testConnectionOnCheckout= true
and for more checking increase checkout timeout as following:
c3p0.checkoutTimeout = 0
and test another test options in C3P0 such as :
connectionTesterClassName
idleConnectionTestPeriod
automaticTestTable
preferredTestQuery
Another way for test is writing a sample code and connect to database by a JDBC pure codes.(by DriverManager and Connection and etc )