Oracle JDBC intermittent Connection Issue - java

I am experiencing a very strange problem
This is a very simple use of JDBC connecting to an Oracle database
OS: Ubuntu
Java Version: 1.5.0_16-b02
1.6.0_17-b04
Database: Oracle 11g Release 11.1.0.6.0
When I make use of the jar file
OJDBC14.jar it connects to the database everytime
When I make use of the jar file
OJDBC5.jar it connects some times and other times it throws an error ( shown below)
If I recompile with Java 6 and use
OJDBC6.jar I get the same results as OJDBC5.jar
I need specific features in JODB5.jar that are not available in OJDBC14.jar
Any ideas
Error
> Connecting to oracle
java.sql.SQLException: Io exception: Connection reset
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:74)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:110)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:171)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:227)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:494)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:474)
at java.sql.DriverManager.getConnection(DriverManager.java:525)
at java.sql.DriverManager.getConnection(DriverManager.java:171)
at TestConnect.main(TestConnect.java:13)
Code
Below is the code I am using
import java.io.*;
import java.sql.*;
public class TestConnect {
public static void main(String[] args) {
try {
System.out.println("Connecting to oracle");
Connection con=null;
Class.forName("oracle.jdbc.driver.OracleDriver");
con=DriverManager.getConnection(
"jdbc:oracle:thin:#172.16.48.100:1535:sample",
"JOHN",
"90009000");
System.out.println("Connected to oracle");
con.close();
System.out.println("Goodbye");
} catch(Exception e) { e.printStackTrace(); }
}
}

There is a solution provided to this problem in some of the OTN forums (https://kr.forums.oracle.com/forums/thread.jspa?messageID=3699989). But, the root cause of the problem is not explained. Following is my attempt to explain the root cause of the problem.
The Oracle JDBC drivers communicate with the Oracle server in a secure way. The drivers use the java.security.SecureRandom class to gather entropy for securing the communication. This class relies on the native platform support for gathering the entropy.
Entropy is the randomness collected/generated by an operating system or application for use in cryptography or other uses that require random data. This randomness is often collected from hardware sources, either from the hardware noises, audio data, mouse movements or specially provided randomness generators. The kernel gathers the entropy and stores it is an entropy pool and makes the random character data available to the operating system processes or applications through the special files /dev/random and /dev/urandom.
Reading from /dev/random drains the entropy pool with requested amount of bits/bytes, providing a high degree of randomness often desired in cryptographic operations. In case, if the entropy pool is completely drained and sufficient entropy is not available, the read operation on /dev/random blocks until additional entropy is gathered. Due to this, applications reading from /dev/random may block for some random period of time.
In contrast to the above, reading from the /dev/urandom does not block. Reading from /dev/urandom, too, drains the entropy pool but when short of sufficient entropy, it does not block but reuses the bits from the partially read random data. This is said to be susceptible to cryptanalytical attacks. This is a theorotical possibility and hence it is discouraged to read from /dev/urandom to gather randomness in cryptographic operations.
The java.security.SecureRandom class, by default, reads from the /dev/random file and hence sometimes blocks for random period of time. Now, if the read operation does not return for a required amount of time, the Oracle server times out the client (the jdbc drivers, in this case) and drops the communication by closing the socket from its end. The client when tries to resume the communication after returning from the blocking call encounters the IO exception. This problem may occur randomly on any platform, especially, where the entropy is gathered from hardware noises.
As suggested in the OTN forum, the solution to this problem is to override the default behaviour of java.security.SecureRandom class to use the non-blocking read from /dev/urandom instead of the blocking read from /dev/random. This can be done by adding the following system property -Djava.security.egd=file:///dev/urandom to the JVM. Though this is a good solution for the applications like the JDBC drivers, it is discouraged for applications that perform core cryptographic operations like crytographic key generation.
Other solutions could be to use different random seeder implementations available for the platform that do not rely on hardware noises for gathering entropy. With this, you may still require to override the default behaviour of java.security.SecureRandom.
Increasing the socket timeout on the Oracle server side can also be a solution but the side effects should be assessed from the server point of view before attempting this.

I was facing exactly the same problem. With Windows Vista I could not reproduce the problem but on Ubuntu I reproduced the 'connection reset'-Error constantly.
I found
http://forums.oracle.com/forums/thread.jspa?threadID=941911&tstart=0&messageID=3793101
According to a user on that forum:
I opened a ticket with Oracle and this is what they told me.
java.security.SecureRandom is a standard API provided by sun. Among
various methods offered by this class void nextBytes(byte[]) is one.
This method is used for generating random bytes. Oracle 11g JDBC
drivers use this API to generate random number during login. Users
using Linux have been encountering SQLException("Io exception:
Connection reset").
The problem is two fold
The JVM tries to list all the files in the /tmp (or alternate tmp directory set by -Djava.io.tmpdir) when SecureRandom.nextBytes(byte[])
is invoked. If the number of files is large the method takes a long
time to respond and hence cause the server to timeout
The method void nextBytes(byte[]) uses /dev/random on Linux and on some machines which lack the random number generating hardware the
operation slows down to the extent of bringing the whole login process
to a halt. Ultimately the the user encounters SQLException("Io
exception: Connection reset")
Users upgrading to 11g can encounter this issue if the underlying OS
is Linux which is running on a faulty hardware.
Cause The cause of this has not yet been determined exactly. It could
either be a problem in your hardware or the fact that for some reason
the software cannot read from dev/random
Solution Change the setup for your application, so you add the next
parameter to the java command:
-Djava.security.egd=file:/dev/../dev/urandom
We made this change in our java.security file and it has gotten rid of
the error.
which solved my problem.

A "connection reset" error message generally means that the other side has aborted the connection during the attempt to create a connection (the handshake). This has a lot of possible causes. A bug in the JDBC driver, a timeout at the DB side, a restart of the database, the DB being run out of available connections, poor network quality, bad virusscanner/firewall/proxy, etc.
As it happens intermittely, a bug in the JDBC driver can be less or more excluded. Left behind the remaining possible causes. I suggest to start with looking in the logs of the DB server.

It's hard to say, but if I would check the actual version of the JDBC driver. Make sure it's 11.1.0.6.
Oracle doesn't include the database version in the filename. So the driver for 11.2 is the exact same name as the driver for 11.1 - ojdbc5.jar. I would extract the driver jar file, and find the MANIFEST.MF file, this will contain some version information. Make sure the version of the JDBC driver matches the version of your database. I suspect it may be a version issue, since there isn't a jar file named ojdbc14.jar on Oracle's 11.1.0.6 download page.
If the version matches - I'm out of ideas :)

Other thing that was causing me this problem was having the HOSTNAME settings wrong. My connection attempt was hanged at:
"main" prio=10 tid=0x00007f7cc8009000 nid=0x2f3a runnable [0x00007f7cce69e000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.getLocalHostName(Native Method)
at java.net.InetAddress.getLocalHost(InetAddress.java:1444)
at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:176)
at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:162)
at java.security.AccessController.doPrivileged(Native Method)
So make sure you have an entry for your hostname in /etc/hosts/.
If you issue a hostname command like this:
$ hostname
my.server.com
You need a line in your /etc/hosts:
127.0.0.1 my my.server.com

As per Bug https://bugs.openjdk.java.net/browse/JDK-6202721
Java will not consder -Djava.security.egd=file:/dev/urandom
It should be -Djava.security.egd=file:/dev/./urandom

Just to clarify - at least from what we found on our side!
It is an issue with the setup of the randomizer for Linux in the JDK distribution - and we found it in Java6, not sure about Java7.
The syntax for linux for the randomizer is file:///dev/urandom, but the entry in the file is (probably left/copied from Windows) as file:/dev/urandom.
So then Java probably falls back on the default, which happens to be /dev/random. And which doesn't work on a headless machine!!!

The root cause of this problem has to do with user authentication versions. For each database user, multiple password verifiers are kept in the database. Typically when you upgrade your database, a new password verifier will be added to the list, a stronger one. The following query shows the password verifier versions that are available for each user. For example:
SQL> SELECT PASSWORD_VERSIONS FROM DBA_USERS WHERE USERNAME='SCOTT';
PASSWORD_VERSIONS
-----------------
11G 12C
When upgrading to a newer driver you can use a newer version of the verifier because the driver and server negotiate the strongest possible verifier to to be used. This newer version of the verifier will be more secure and will involve generating larger random numbers or using more complex hashing functions which can explain why you see issues while establishing JDBC connections. As mentioned by other responses using /dev/urandom normally resolves these issues. You can also decide to downgrade your password verifier and make the newer driver use the same older password verifier that your previous driver was using. For example if you want to use the 10G password verifier (for testing purposes only), first you need to make sure it's available for your user.
Set SQLNET.ALLOWED_LOGON_VERSION_SERVER=8 in sqlnet.ora on the server. Then:
SQL> alter user scott identified by "tiger";
User altered.
SQL> SELECT PASSWORD_VERSIONS FROM DBA_USERS WHERE USERNAME='SCOTT';
PASSWORD_VERSIONS
-----------------
10G 11G 12C
Then you can force the JDBC thin driver to use the 10G verifier by setting this JDBC property oracle.jdbc.thinLogonCapability="o3". If you run into the error "ORA-28040: No matching authentication protocol" then that means your server is not allowing the 10G verifier to be used. If that's the case then you need to check your configuration again.

Note that the suggested solution of using /dev/urandom did work the first time for me but didn't work always after that.
DBA at my firm switched of 'SQL* net banners' and that fixed it permanently for me with or without the above.
I don't know what 'SQL* net banners' are, but am hoping by putting this information here that if you have(are) a DBA he(you) would know what to do.

Disabling SQL Net Banners saved us.

-Djava.security.egd=file:/dev/./urandom should be right! not -Djava.security.egd=file:/dev/../dev/urandom or -Djava.security.egd=file:///dev/urandom

I faced the same problem when a liquibase was executed from jenkins. Sporadically this error was thrown to the output and the liquibase change logs were not executed at all.
Solution provided: In the jenkin's maven project, the jdk was updated from jdk8-131 to any newer version (eg java8-162).

OracleXETNSListener - this service has to be started if it was disabled.
run -> services.msc
and look out for that services

Related

Microsoft SQL Server - Query with huge results, causes ASYNC_NETWORK_IO wait issue

I have a Java application requesting about 2.4 million records from a Microsoft SQL Server (Microsoft SQL Server 2008 R2 (SP3))
The application runs fine on all hosts, except one. On this host, the application is able to retrieve data on some occasions. But on some others, it hangs.
Monitoring the MS Sql server indicates that the SPID associated with the query is in an ASYNC_NETWORK_IO wait state.
There are a few links online that talk about it.
https://blogs.msdn.microsoft.com/joesack/2009/01/08/troubleshooting-async_network_io-networkio/
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/6db233d5-8892-4f8a-88c7-b72d0fc59ca9/very-high-asyncnetworkio?forum=sqldatabaseengine
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/1df2cab8-33ca-4870-9daf-ed333a64630c/network-packet-size-and-delay-by-sql-server-sending-data-to-client?forum=sqldatabaseengine
Based on the above, the ASYNC_NETWORK_IO means 2 things:
1. Application is slow to process the results
2. Network between application and DB has some issues.
For #1 above, We analyzed using tcpdumps and found that in the cases where the query goes into ASYNC_NETWORK_IO state, the application server's tcp connection has a window size that oscillates between 0 and a small number, and eventually remains stuck at 0. Based on some more analysis, aspects related to firewalls between DB and application have also been mostly ruled out.
So I am staring at #2, unable to understand what could possibly go wrong. All the more baffling because the same code has been running under similar data loads for more than a year now. And it also runs fine on other hosts.
The JDBC driver being used is sqljdbc4-4.0.jar.
This by default has an adaptive buffering feature, which does things under the hood to reduce application resources.
We use the default fetch size of 128 (which i believe is not a good one).
So i am going to experiment overriding the default adaptive buffering behavior, though the MS docs suggest that it is good to have adaptive buffering for large result sets.
I will change the connection setting to use selectMethod=cursor.
And also change the fetchSize to 1024.
Now if it does not work:
What are some aspects of the problem that are worth investigating.
Assuming its still an issue with the client, what other connection settings, network settings should be inspected/changed to make progress?
If it does work consistently, what is the impact of making the connection setting change to selectMethod=cursor
On the application side?
Database side?
Update: I tested the application adding the selectMethod=cursor to the connection. However, it results in the same issue as above.
Based on discussions with other administrators in the team - at this point the issue may be in the jdbc driver, or on the OS (when it tries to handle the data on the network).
After a good amount of discussions with the System admin, Network Admin and Database admin - it was agreed that somewhere in the OS -> Application stack, the data from network wasn't handled. In the meantime, we tested out a solution where we broke down the query to return smaller sized results. So we broke it down to 5 queries, each returning about 500k records.
Now when we ran these queries sequentially, we still ran into the same issue.
However, when we ran the queries in parallel, it always was successful.
Given that the solution worked always we haven't bothered getting to the root cause of the problem anymore.
On another note, the hardware and software running the application was also outdated. It was running Red Hat 5. So, it could well have to do something with that.

Connecting to old Pervasive DB with JDBC

I'm trying to write a small program for CRUD operations against an old Pervasive 9 database, but as soon as I try to get a connection with Drivermanager.getConnection it throws
"java.io.IOException: LNA session closed"
I've read THAT post but copy/paste on that code didn't help me. Could be it configuration on that Pervasive 9 machine? Is it JDBC10 driver that won't play nice with older versions?
Or.. is it me being stupid? (everything is always possible)
Regards
According to Pervasive, using a newer client to an older engine is not recommended. This is documented at http://docs.pervasive.com/products/database/psqlv11/wwhelp/wwhimpl/js/html/wwhelp.htm#href=getstart/installprep.02.6.html#149125.
That doesn't mean it won't work but there's no guarantee. There are reports of the Btrieve interface working properly but the relational (SQL, ODBC, JDBC) side fails more often mainly because it changes between major versions where the Btrieve side stays the same.
I would suggest using the v9 client (and JDBC driver) or upgrading the server to v10.
#elwis your trouble it's the conecction with the database. I recommend you, make a connection using a reverse ssh tunnel against the machine where you have the database first with port by default of pervasive.
Something like: ssh user#yourmachine -L 15831:192.1X.X0.X80:1583
Then, using the tool can generate the connection DBeaver to see and do what you need.
That works for me.
Good luck.

mysql jdbc communication exception

I have a Java application which initially reads 3 lakhs of data from my MYSQL database.Then it calls an API using an ExecutorService with newFixedThreadPool size=20.
After getting the response from the API it is inserting the responses to my DB.It is working fine for first 2000 rows(nearby).After that I am getting an error like following.
SQLError-com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The
driver was unable to create a connection due to an inability to
establish the client portion of a socket.
This is usually caused by a limit on the number of sockets imposed by
the operating system. This limit is usually configurable.
For Unix-based platforms, see the manual page for the 'ulimit'
command. Kernel or system reconfiguration may also be required.
For Windows-based platforms, see Microsoft Knowledge Base Article
196271 (Q196271).
Anyone could help me to fix this issue?
I was able to fix this problem by increasing the # of sockets that can be opened in Windows:
From the Windows Start menu, run the regedit.exe application
In the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters directory, create a new DWORD value named MaxUserPort with a decimal value of 65,000 (the default is 5,000)
After closing the regedit application, restart your computer.
(See also Increasing the number of Windows sockets or ports, Maximum Socket Connections)
A note of caution: When an application is using more than 5,000 socket connections, that may be an indication that system resources are not being used in a sustainable way. It would be good to investigate the root cause for why so many sockets are being opened simultaneously.

What mechanisms are used by SQL Server 2008 Mirroring to notify a driver that the primary has failed over?

I've got a database that is mirrored using SQL 2008 Mirroring.
I have a java application, running on Linux, using the Microsoft SQL
type 4 JDBC drivers.
I have this setup duplicated as a QA environment.
On my QA environment, when I manually fail over the database,
providing a successful connection had already been made, the failover
was completely transparent. I did not have to implement anything in
order to get the application to talk to the new Principle (Old
mirror).
In the live environment however, the connections stop working once I
have manually failed over.
There are quite a few things different between the live and QA environments, but not anything I'd consider fundamental to this process, communication between databases is all very localised in both situations and there are no firewalls (Except the ones built into Windows Server 2008) between my java app and the Windows boxes running SQL 2008.
Does anyone have any ideas about how I can go about diagnosing this issue? Or can anyone tell me how this failover transparency occurs so I can work out how to diagnose this myself?
This behaviour is handled by the SQL Server provider as part of the Client Redirect behaviour. This was part of the SQL Server 2005 JDBC (Java Database Connectivity) 1.1 Driver, I'm unsure if that provider was directly used for the new type 4 drivers though?
In your live environment, is your application successfully connecting to the primary server before it fails over, so it is able to cache the failover partner and make use of that when the failover occurs?
To be sure, you can explicitly state the failover partner in the connection string, which is the recommended practice:
jdbc:sqlserver://serverA:1433; databaseName=AdventureWorks; integratedSecurity=true; failoverPartner=serverB
The full documentation of the redirect behaviour can be found here.
The addition of the failover documentation here suggests that it could be a manual consideration.

JavaDB connection issues; database not found

I am having a problem with Java DB that I just don't know how to resolve. I am creating a DB and connecting to it using Java DB's native JDBC driver. If I relocate that database physically and try to connect to it using its new path, I consistently get XJ004 errors:
ERROR XJ004: Database 'blahblah' not found.
I am sure I am using the correct connection string. Is there any possibility the DB is somehow getting corrupted? Or is there some encoding of the DB path in the DB such that if you relocate a Java DB it gets confused?
I'm really at a loss here. :( Please help!
Jim
Have you verified that this error message isn't also used when there's no listener on the host machine ... and were you using JavaDB on your local machine before the relocation? Many database systems (and I'm not that familiar with JavaDB) ship set-up to only allow connections from localhost for security reasons. On PostgreSQL for instance, you have to allow TCP connections and bounce the daemon to obtain a remote connection.
Anyway ... since the problem started when you when remote, look for issues related to that first! (And if you can run your application on the remote machine, does that work?)
There must be a file named derby.log somewhere. Check the error there. If it is not detailed enough, try setting derby.stream.error.logSeverityLevel to a lower value. See the manual for more information.

Categories