We have one of the customer java websphere applications pointing to oracle database encountering connection close error while doing the plsql stored procedure execution. Sometimes after a few minutes into the stored procedure call and a few times after ~2 hours into execution. In a normal scenario, this job succeeds, within <10 seconds . So during failure time, it may have been waiting for some resource or other bottleneck that resulted in long execution time. So want to understand if anybody encountered such an error and what is the cause and fix of this?
The database is oracle version 11.2.0.4.
Below is the application log.
com.ibm.websphere.ce.cm.ObjectClosedException: DSRA9110E: Connection is closed.
at com.ibm.ws.rsadapter.jdbc.WSJdbcWrapper.createClosedException(WSJdbcWrapper.java:122)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.activate(WSJdbcConnection.java:2936)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.setAutoCommit(WSJdbcConnection.java:3512)
We found below oracle doc matching the error but it seems if it would have been this bug then rerun also would have been failed. So our case might be different.
Protocol Violation and Connection is Closed exceptions When Using JDBC 12.1.0.2 with WebSphere (Doc ID 2359390.1)
Related
“Communications link failure” error on MySQL using JDBC after the program has run for 38 hrs. Another program on the same machine hitting the same server (also java) continues to run. Both are using the exact same java code to access the server, as do the java programs running on 4 other machines at the same time. I get this error randomly on different machines, after a program has been running successfully (reads/inserts/updates to the database) for quite a while. The error includes the first line above the stack trace:
The last packet successfully received from the server was n milliseconds ago. where n has varied between 0 and 25.
I might suspect that the server has run out connections to hand out, but I get a totally different error when that happens (I encountered it and updated my server to increase max connections, which eliminated that error). I get the error whether I use c3p0 to pool connections, or not.
Please read before flagging as a duplicate. The connect string and database are fine, as it worked for 38 hours (and continues to work in other processes on the same machine and different machines accessing the same MySQL server) using exactly the same java code. This is a transient/random error. If I simply restart the program, it continues successfully. I would say it's OS related, except that it occurs on both Windows and Mac OS X. Which leads me to believe it's likely something going on in the java VM.
Considering the point that you mention, that it is running fine until 38 hours and then you get the following issue - It only leads to one logical point:
If your connection is IDLE for a considerably longer duration. This idle connection would evaluate connection.isClosed(); to TRUE, and when we try to execute the above statement it will then fire the following exception.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
How to remediate this issue?
Switch to DB Connection Pooling. Hope this helps!
We are facing some strange behavior in calling stored procedure from java. We are using plain JDBC connection in our stand alone application to invoke some stored procedure which is taking more than 1 hour to finish.
Problem is that when we are after the stored procedure finishes executing, we are not getting any response in java code. Java code seems to be stuck but after the Socket is getting timed out, we are getting readTimedOut exception in java. This happens only on 1 instance of our database, on other instance it works fine
Is the firewall between the servers may cause the problem. Or there is some parameter in oracle that is causing the connection to be terminated. Is there some parameter in JDBC driver. We are using jdbc.odbc.OracleDrive
I am trying to insert the tens of millions of data into oracle database by through java application but after 850k data inserted I am getting the error:
java.sql.exception: IO Exception :Connection reset by peer socket
write error.
I have a similar issue in my aplication.
In my case, after a few minutes in which the connection has not used, oracle server resets the connection and when I had to use it again, this exception was thrown.
I found that the problem was that oracle sets the timeout for connection at 10 minutes in sqlora.net file.
Maybe you can try to increase this time interval or check if the connection is still valid instead to check if it is closed (a connection can be not valid even if it is not closed).
The last option works fine for me.
I hope this helps.
My java application does use DB Connection pooling. One of the functionality started failing today with this error:
[BEA][SQLServer JDBC Driver]No more data available to read
This doesn't occur daily. Once I restart my application server things look fine for some days and this error comes back again.
Anyone encountered this error? Reasons might vary, but I would like to know those various reasons to mitigate my issue.
Is it possible that the database or network connection has briefly had an outage? You might expect any currently open result sets then to become invalid with resulting errors.
I've never seen this particular error, but then I don't work with BEA or SQL Server, but a quick google does show other folks suggesting such a cause.
When you're using a connection pool, if you do get such a glitch, then all connections in teh pool become "stale" or invalid. My application server (WebSphere) has the option to discard the entire connection pool after particular errors are detected. The result then is that one unlucky request sees the error, but then subsequent requests get a new connection and recover. If you don't discard the whole pool then you get a failure as each stale connection is used and discarded.
I suggest you investigate to see a). whether your app server has such a capability b). how you application responds if the database is bounced, if this replicates the error then maybe you've found the cause.
I'm getting the following excpetion:
java.sql.SQLException: Protocol violation
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:145)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:190)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:286)
at oracle.jdbc.driver.T4C80all.receive(T4C80all.java:766)
at oracle.jdbc.driver.T4CPreparedStatement.do0all8(T4CPreparedStatement.java:216)
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1225)
at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:373)
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:284)
The Oracle system is running 10.2.0.3.0 on Solaris 5.10. The jdbc driver is running on JDK 1.6.0_21 (if it's import the java is running on a Solaris 5.10 machine as well). I've tried several different oracle thin drivers including the latest and the one that appears to exactly match the oracle version.
The query I'm running is fairly simple: "select * from some_table order by key1, key2, key3" Then iterating through the result set and writing to a file. The table has around 12 million rows, so I expect the process is running long, but it seems to die within 5-15 mins into it. Each time I run it, it blows up on a different row, so I don't think the problem is with the data.
I found the oracle alert log but I couldn't tell that anything in there was related to my process. Still, I'm no oracle expert and perhaps there's an oracle setting I need to look at. Strangely enough, I'm running about five of these type of queries (a couple are a bit more complicated) on different connections and only two simplest ones ever get this problem.
Any help or ideas on what to look at to narrow down the problem would be appreciated.
For future googlers who are have to this page, here is the problem we had .
The protocol violation exception was being logged on application logs and Oracle trace.
Oracle trace
This is error from oracle trace files
--- PROTOCOL VIOLATION DETECTED ---
----- Dump Cursor sql_id=1j5kjnkncpp xsc=0x2a053a2a0 cur=0x2a052f1cf0 ---
----- Current SQL Statement for this session (sql_id=1jjns4k6npp) -----
select xyz
From Application Logs
Caused by: org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation; uncategorized SQLException for SQL []; SQL state [72000]; error code [20000];
Symptom
This exception was happening occasionally. The stack trace had different sql in it which was very confusing. Running the sql with sql plus worked fine.
Root Cause
The exception was thrown when oracle driver was trying to export a CLOB data. This was happening with only few records, not all of them. The data as such was a file. Visually we could not make out what was wrong with that data.
Why we were seeing errors in oracle logs ?
So if this was a driver defect, why did we see the error in oracle trace ? Logically the driver errors should be only confined to application logs.
The reasons was that when protocol violation happened, the connection got corrupted. This connection was returned to the connection pool. Any user or job when will use that connection would not work and would experience error.
That is why it will happen at random places, with random users
Solution
A short term fix was to change this property in connection pool. We are using DBCP connection pool.
Changed from
ds.setTestOnBorrow(false);
to
ds.setTestOnBorrow(true);
Now when the pool returns a corrupted connection to the pool, before app borrows this connection , it would test for validity. If connection is unusable, pool would discard and then app gets a new/valid connection.
If you enable connection pool logs, you should see the exception which normally is swallowed.
Driver Upgrade
Upgrade to OJDBC 12.1.0.2 from OJDBC 12.1.0.1 solved the problem, even for the problematic rows.
Some other links for reference
https://confluence.atlassian.com/display/CONFKB/java.sql.SQLException%3A+Protocol+violation+caught+while+accessing+a+page+and+Oracle+DB+is+used
Apparently adding -d64 to the java command line fixes this problem. Looks like a Solaris 64-bit issue.
In my case, using tomcat 8.5 (standard connection pool), oracle 19, when oracle emits an warning message (like 'your password will expire in n days'), the java connection object interprets like an error.
I just change the PASSWORD, by the same name, and the problem was solved.