We are facing some strange behavior in calling stored procedure from java. We are using plain JDBC connection in our stand alone application to invoke some stored procedure which is taking more than 1 hour to finish.
Problem is that when we are after the stored procedure finishes executing, we are not getting any response in java code. Java code seems to be stuck but after the Socket is getting timed out, we are getting readTimedOut exception in java. This happens only on 1 instance of our database, on other instance it works fine
Is the firewall between the servers may cause the problem. Or there is some parameter in oracle that is causing the connection to be terminated. Is there some parameter in JDBC driver. We are using jdbc.odbc.OracleDrive
Related
We have one of the customer java websphere applications pointing to oracle database encountering connection close error while doing the plsql stored procedure execution. Sometimes after a few minutes into the stored procedure call and a few times after ~2 hours into execution. In a normal scenario, this job succeeds, within <10 seconds . So during failure time, it may have been waiting for some resource or other bottleneck that resulted in long execution time. So want to understand if anybody encountered such an error and what is the cause and fix of this?
The database is oracle version 11.2.0.4.
Below is the application log.
com.ibm.websphere.ce.cm.ObjectClosedException: DSRA9110E: Connection is closed.
at com.ibm.ws.rsadapter.jdbc.WSJdbcWrapper.createClosedException(WSJdbcWrapper.java:122)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.activate(WSJdbcConnection.java:2936)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.setAutoCommit(WSJdbcConnection.java:3512)
We found below oracle doc matching the error but it seems if it would have been this bug then rerun also would have been failed. So our case might be different.
Protocol Violation and Connection is Closed exceptions When Using JDBC 12.1.0.2 with WebSphere (Doc ID 2359390.1)
We've been trying to figure out a way to run a very long running (3-4 hours) stored procedure remotely using either Sybase's isql client or Java. The stored procedure starts fine and even runs and returns some print output. But then hits this very large select * into #tmp .... statement that takes over 3 hours to run. But somewhere after about 2 hours, the network connection is dropped...most likely due to in-activity as no print output comes back from the stored procedure and the isql client is not sending any requests.
Background : When we run this stored procedure remotely from a Linux batch server using the isql command line client, it starts, but then because of the length of time (3-4 hours), something in our network layers simply drops the connection and the Sybase server never responds back to the client. Same issue when we try and run the stored procedure using a GUI client like DBVisualizer on our laptops...remote execution starts, runs, but never responds. Basically, any attempt to run the stored procedure remotely fails.
Non-network : However, when we run the stored procedure ON the actual Sybase Unix server (I.e. the Sybase database and the isql client are on the same Unix server) using the isql client it DOES finish. So, when running the stored procedure in a "local" non-remote situation it finishes fine.
Conclusion : That something in the network layers simply drops the connection because of in-activity.
That said, We have tried the following without success :
a.) Changed the Linux tcp_keepalive to 4 hours on the batch server.
b.) Changed the Unix/AIX tcp_keepidle to 4 hours on the Sybase server.
c.) Wrote a custom Java multi-thread process that created 2 threaded SQL calls, one for the Stored Procedure, one for a simple "keep-alive" select. But once the stored proc starts running it locks/blocks out the other thread. This appears to be a sympthom of the Sybase JDBC Drivers, they simply do NOT allow 2 SQL statements to run in parallel... Concurrent use of same JDBC connection by multiple threads and Is asynchronous jdbc call possible?.
My question is, is there any other options I've over-looked? We've been told by our Database admins that the Sybase engine has no limitations or timeouts set, but is there anyway to confirm this? We are currently checking our firewalls, but i'm skeptical that a firewall maintains a "session"...although I'm not a firewall expert. Again, is there anyway to test a firewall timeout?
tia, adym
UPDATE : 09/11/2017 -
I set the DEFAULT_QUERY_TIMEOUT to 8 hours (I think). From what I read, you provide this in seconds. But this made no difference.
Properties props = new Properties();
props.put ( "user", getJDBCUsername() );
props.put ( "password", getJDBCPassword(isEncrypted()) );
props.put ( "DEFAULT_QUERY_TIMEOUT", 28800 );
this.connection = DriverManager.getConnection ( getJDBCUrl (), props );
We are getting the exception of java.sql.SQLRecoverableException: No more data to read from socket in our application, which is making the connection corrupted. After this, we are not able to fetch any connection from the connection pool and it is displaying the error as Connection closed. Now we are mitigating this issue by restarting the server. We found the root cause of this issue in one area of our application, we are calling a procedure which is calling an external db link and if that external db is down, we get this 'java.sql.SQLRecoverableException: No more data to read from socket` which is an expected behavior.
Here my problem is: As this procedure is calling from only a part of the application, it is affecting the whole connections in the connection pool. Is there a way to handle this exception in that specific part (where this exception is thrown) so that this should not affect the connection pool?
Note: Code for closing the connection is properly implemented in the code.
“Communications link failure” error on MySQL using JDBC after the program has run for 38 hrs. Another program on the same machine hitting the same server (also java) continues to run. Both are using the exact same java code to access the server, as do the java programs running on 4 other machines at the same time. I get this error randomly on different machines, after a program has been running successfully (reads/inserts/updates to the database) for quite a while. The error includes the first line above the stack trace:
The last packet successfully received from the server was n milliseconds ago. where n has varied between 0 and 25.
I might suspect that the server has run out connections to hand out, but I get a totally different error when that happens (I encountered it and updated my server to increase max connections, which eliminated that error). I get the error whether I use c3p0 to pool connections, or not.
Please read before flagging as a duplicate. The connect string and database are fine, as it worked for 38 hours (and continues to work in other processes on the same machine and different machines accessing the same MySQL server) using exactly the same java code. This is a transient/random error. If I simply restart the program, it continues successfully. I would say it's OS related, except that it occurs on both Windows and Mac OS X. Which leads me to believe it's likely something going on in the java VM.
Considering the point that you mention, that it is running fine until 38 hours and then you get the following issue - It only leads to one logical point:
If your connection is IDLE for a considerably longer duration. This idle connection would evaluate connection.isClosed(); to TRUE, and when we try to execute the above statement it will then fire the following exception.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
How to remediate this issue?
Switch to DB Connection Pooling. Hope this helps!
I'm getting the following excpetion:
java.sql.SQLException: Protocol violation
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:145)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:190)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:286)
at oracle.jdbc.driver.T4C80all.receive(T4C80all.java:766)
at oracle.jdbc.driver.T4CPreparedStatement.do0all8(T4CPreparedStatement.java:216)
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1225)
at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:373)
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:284)
The Oracle system is running 10.2.0.3.0 on Solaris 5.10. The jdbc driver is running on JDK 1.6.0_21 (if it's import the java is running on a Solaris 5.10 machine as well). I've tried several different oracle thin drivers including the latest and the one that appears to exactly match the oracle version.
The query I'm running is fairly simple: "select * from some_table order by key1, key2, key3" Then iterating through the result set and writing to a file. The table has around 12 million rows, so I expect the process is running long, but it seems to die within 5-15 mins into it. Each time I run it, it blows up on a different row, so I don't think the problem is with the data.
I found the oracle alert log but I couldn't tell that anything in there was related to my process. Still, I'm no oracle expert and perhaps there's an oracle setting I need to look at. Strangely enough, I'm running about five of these type of queries (a couple are a bit more complicated) on different connections and only two simplest ones ever get this problem.
Any help or ideas on what to look at to narrow down the problem would be appreciated.
For future googlers who are have to this page, here is the problem we had .
The protocol violation exception was being logged on application logs and Oracle trace.
Oracle trace
This is error from oracle trace files
--- PROTOCOL VIOLATION DETECTED ---
----- Dump Cursor sql_id=1j5kjnkncpp xsc=0x2a053a2a0 cur=0x2a052f1cf0 ---
----- Current SQL Statement for this session (sql_id=1jjns4k6npp) -----
select xyz
From Application Logs
Caused by: org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation; uncategorized SQLException for SQL []; SQL state [72000]; error code [20000];
Symptom
This exception was happening occasionally. The stack trace had different sql in it which was very confusing. Running the sql with sql plus worked fine.
Root Cause
The exception was thrown when oracle driver was trying to export a CLOB data. This was happening with only few records, not all of them. The data as such was a file. Visually we could not make out what was wrong with that data.
Why we were seeing errors in oracle logs ?
So if this was a driver defect, why did we see the error in oracle trace ? Logically the driver errors should be only confined to application logs.
The reasons was that when protocol violation happened, the connection got corrupted. This connection was returned to the connection pool. Any user or job when will use that connection would not work and would experience error.
That is why it will happen at random places, with random users
Solution
A short term fix was to change this property in connection pool. We are using DBCP connection pool.
Changed from
ds.setTestOnBorrow(false);
to
ds.setTestOnBorrow(true);
Now when the pool returns a corrupted connection to the pool, before app borrows this connection , it would test for validity. If connection is unusable, pool would discard and then app gets a new/valid connection.
If you enable connection pool logs, you should see the exception which normally is swallowed.
Driver Upgrade
Upgrade to OJDBC 12.1.0.2 from OJDBC 12.1.0.1 solved the problem, even for the problematic rows.
Some other links for reference
https://confluence.atlassian.com/display/CONFKB/java.sql.SQLException%3A+Protocol+violation+caught+while+accessing+a+page+and+Oracle+DB+is+used
Apparently adding -d64 to the java command line fixes this problem. Looks like a Solaris 64-bit issue.
In my case, using tomcat 8.5 (standard connection pool), oracle 19, when oracle emits an warning message (like 'your password will expire in n days'), the java connection object interprets like an error.
I just change the PASSWORD, by the same name, and the problem was solved.