I have a Java application which initially reads 3 lakhs of data from my MYSQL database.Then it calls an API using an ExecutorService with newFixedThreadPool size=20.
After getting the response from the API it is inserting the responses to my DB.It is working fine for first 2000 rows(nearby).After that I am getting an error like following.
SQLError-com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The
driver was unable to create a connection due to an inability to
establish the client portion of a socket.
This is usually caused by a limit on the number of sockets imposed by
the operating system. This limit is usually configurable.
For Unix-based platforms, see the manual page for the 'ulimit'
command. Kernel or system reconfiguration may also be required.
For Windows-based platforms, see Microsoft Knowledge Base Article
196271 (Q196271).
Anyone could help me to fix this issue?
I was able to fix this problem by increasing the # of sockets that can be opened in Windows:
From the Windows Start menu, run the regedit.exe application
In the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters directory, create a new DWORD value named MaxUserPort with a decimal value of 65,000 (the default is 5,000)
After closing the regedit application, restart your computer.
(See also Increasing the number of Windows sockets or ports, Maximum Socket Connections)
A note of caution: When an application is using more than 5,000 socket connections, that may be an indication that system resources are not being used in a sustainable way. It would be good to investigate the root cause for why so many sockets are being opened simultaneously.
Related
I have a Java application requesting about 2.4 million records from a Microsoft SQL Server (Microsoft SQL Server 2008 R2 (SP3))
The application runs fine on all hosts, except one. On this host, the application is able to retrieve data on some occasions. But on some others, it hangs.
Monitoring the MS Sql server indicates that the SPID associated with the query is in an ASYNC_NETWORK_IO wait state.
There are a few links online that talk about it.
https://blogs.msdn.microsoft.com/joesack/2009/01/08/troubleshooting-async_network_io-networkio/
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/6db233d5-8892-4f8a-88c7-b72d0fc59ca9/very-high-asyncnetworkio?forum=sqldatabaseengine
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/1df2cab8-33ca-4870-9daf-ed333a64630c/network-packet-size-and-delay-by-sql-server-sending-data-to-client?forum=sqldatabaseengine
Based on the above, the ASYNC_NETWORK_IO means 2 things:
1. Application is slow to process the results
2. Network between application and DB has some issues.
For #1 above, We analyzed using tcpdumps and found that in the cases where the query goes into ASYNC_NETWORK_IO state, the application server's tcp connection has a window size that oscillates between 0 and a small number, and eventually remains stuck at 0. Based on some more analysis, aspects related to firewalls between DB and application have also been mostly ruled out.
So I am staring at #2, unable to understand what could possibly go wrong. All the more baffling because the same code has been running under similar data loads for more than a year now. And it also runs fine on other hosts.
The JDBC driver being used is sqljdbc4-4.0.jar.
This by default has an adaptive buffering feature, which does things under the hood to reduce application resources.
We use the default fetch size of 128 (which i believe is not a good one).
So i am going to experiment overriding the default adaptive buffering behavior, though the MS docs suggest that it is good to have adaptive buffering for large result sets.
I will change the connection setting to use selectMethod=cursor.
And also change the fetchSize to 1024.
Now if it does not work:
What are some aspects of the problem that are worth investigating.
Assuming its still an issue with the client, what other connection settings, network settings should be inspected/changed to make progress?
If it does work consistently, what is the impact of making the connection setting change to selectMethod=cursor
On the application side?
Database side?
Update: I tested the application adding the selectMethod=cursor to the connection. However, it results in the same issue as above.
Based on discussions with other administrators in the team - at this point the issue may be in the jdbc driver, or on the OS (when it tries to handle the data on the network).
After a good amount of discussions with the System admin, Network Admin and Database admin - it was agreed that somewhere in the OS -> Application stack, the data from network wasn't handled. In the meantime, we tested out a solution where we broke down the query to return smaller sized results. So we broke it down to 5 queries, each returning about 500k records.
Now when we ran these queries sequentially, we still ran into the same issue.
However, when we ran the queries in parallel, it always was successful.
Given that the solution worked always we haven't bothered getting to the root cause of the problem anymore.
On another note, the hardware and software running the application was also outdated. It was running Red Hat 5. So, it could well have to do something with that.
i am currently testing a server with an automatic test client that simulates a large number of users. Both the server and the client are written in Java. The client opens a tcp/ip connection for every user. Both the server and client run on Ubuntu linux, client runs on 11.04 and server on 10.04.
The testing went good up till 27000 concurrently open connections, after that i decided to jump to 36000 (the servers and clients resources weren't really all that used up at 27000 so i decided to make a slightly bigger jump). When i tried running the test for 36k i got the following exception on the client side:
java.net.BindException: cannot assign requested address
As far as i know at 36k i should still have free ports since not much else is running on either machine and tcp limits the port number at 2^16 which is 65536. Now since it is linux i also set the number of open files for the user to 100k with ulimit -n 100000.
But i am still getting the same exception.
I'm wondering what else could be a possible cause for the mentioned exception, or does linux in some other way limit the number of outgoing connections ?
Thanks in advance,
Danijel
By default, Linux picks dynamically assigned ports from the range 32768..61000. The others are available for static assignment, if you bind to a specific port number. The range can be changed if you want more of the ports to be available for dynamic assignment, but just be careful that you do not include ports that are used for specific services that you need (e.g. 6000 for X11). Also you should not allow ports < 1024 to be dynamically assigned since they are privileged. To check or change the range:
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
# echo "16384 65535" > /proc/sys/net/ipv4/ip_local_port_range
I am experiencing a very strange problem
This is a very simple use of JDBC connecting to an Oracle database
OS: Ubuntu
Java Version: 1.5.0_16-b02
1.6.0_17-b04
Database: Oracle 11g Release 11.1.0.6.0
When I make use of the jar file
OJDBC14.jar it connects to the database everytime
When I make use of the jar file
OJDBC5.jar it connects some times and other times it throws an error ( shown below)
If I recompile with Java 6 and use
OJDBC6.jar I get the same results as OJDBC5.jar
I need specific features in JODB5.jar that are not available in OJDBC14.jar
Any ideas
Error
> Connecting to oracle
java.sql.SQLException: Io exception: Connection reset
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:74)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:110)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:171)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:227)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:494)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:474)
at java.sql.DriverManager.getConnection(DriverManager.java:525)
at java.sql.DriverManager.getConnection(DriverManager.java:171)
at TestConnect.main(TestConnect.java:13)
Code
Below is the code I am using
import java.io.*;
import java.sql.*;
public class TestConnect {
public static void main(String[] args) {
try {
System.out.println("Connecting to oracle");
Connection con=null;
Class.forName("oracle.jdbc.driver.OracleDriver");
con=DriverManager.getConnection(
"jdbc:oracle:thin:#172.16.48.100:1535:sample",
"JOHN",
"90009000");
System.out.println("Connected to oracle");
con.close();
System.out.println("Goodbye");
} catch(Exception e) { e.printStackTrace(); }
}
}
There is a solution provided to this problem in some of the OTN forums (https://kr.forums.oracle.com/forums/thread.jspa?messageID=3699989). But, the root cause of the problem is not explained. Following is my attempt to explain the root cause of the problem.
The Oracle JDBC drivers communicate with the Oracle server in a secure way. The drivers use the java.security.SecureRandom class to gather entropy for securing the communication. This class relies on the native platform support for gathering the entropy.
Entropy is the randomness collected/generated by an operating system or application for use in cryptography or other uses that require random data. This randomness is often collected from hardware sources, either from the hardware noises, audio data, mouse movements or specially provided randomness generators. The kernel gathers the entropy and stores it is an entropy pool and makes the random character data available to the operating system processes or applications through the special files /dev/random and /dev/urandom.
Reading from /dev/random drains the entropy pool with requested amount of bits/bytes, providing a high degree of randomness often desired in cryptographic operations. In case, if the entropy pool is completely drained and sufficient entropy is not available, the read operation on /dev/random blocks until additional entropy is gathered. Due to this, applications reading from /dev/random may block for some random period of time.
In contrast to the above, reading from the /dev/urandom does not block. Reading from /dev/urandom, too, drains the entropy pool but when short of sufficient entropy, it does not block but reuses the bits from the partially read random data. This is said to be susceptible to cryptanalytical attacks. This is a theorotical possibility and hence it is discouraged to read from /dev/urandom to gather randomness in cryptographic operations.
The java.security.SecureRandom class, by default, reads from the /dev/random file and hence sometimes blocks for random period of time. Now, if the read operation does not return for a required amount of time, the Oracle server times out the client (the jdbc drivers, in this case) and drops the communication by closing the socket from its end. The client when tries to resume the communication after returning from the blocking call encounters the IO exception. This problem may occur randomly on any platform, especially, where the entropy is gathered from hardware noises.
As suggested in the OTN forum, the solution to this problem is to override the default behaviour of java.security.SecureRandom class to use the non-blocking read from /dev/urandom instead of the blocking read from /dev/random. This can be done by adding the following system property -Djava.security.egd=file:///dev/urandom to the JVM. Though this is a good solution for the applications like the JDBC drivers, it is discouraged for applications that perform core cryptographic operations like crytographic key generation.
Other solutions could be to use different random seeder implementations available for the platform that do not rely on hardware noises for gathering entropy. With this, you may still require to override the default behaviour of java.security.SecureRandom.
Increasing the socket timeout on the Oracle server side can also be a solution but the side effects should be assessed from the server point of view before attempting this.
I was facing exactly the same problem. With Windows Vista I could not reproduce the problem but on Ubuntu I reproduced the 'connection reset'-Error constantly.
I found
http://forums.oracle.com/forums/thread.jspa?threadID=941911&tstart=0&messageID=3793101
According to a user on that forum:
I opened a ticket with Oracle and this is what they told me.
java.security.SecureRandom is a standard API provided by sun. Among
various methods offered by this class void nextBytes(byte[]) is one.
This method is used for generating random bytes. Oracle 11g JDBC
drivers use this API to generate random number during login. Users
using Linux have been encountering SQLException("Io exception:
Connection reset").
The problem is two fold
The JVM tries to list all the files in the /tmp (or alternate tmp directory set by -Djava.io.tmpdir) when SecureRandom.nextBytes(byte[])
is invoked. If the number of files is large the method takes a long
time to respond and hence cause the server to timeout
The method void nextBytes(byte[]) uses /dev/random on Linux and on some machines which lack the random number generating hardware the
operation slows down to the extent of bringing the whole login process
to a halt. Ultimately the the user encounters SQLException("Io
exception: Connection reset")
Users upgrading to 11g can encounter this issue if the underlying OS
is Linux which is running on a faulty hardware.
Cause The cause of this has not yet been determined exactly. It could
either be a problem in your hardware or the fact that for some reason
the software cannot read from dev/random
Solution Change the setup for your application, so you add the next
parameter to the java command:
-Djava.security.egd=file:/dev/../dev/urandom
We made this change in our java.security file and it has gotten rid of
the error.
which solved my problem.
A "connection reset" error message generally means that the other side has aborted the connection during the attempt to create a connection (the handshake). This has a lot of possible causes. A bug in the JDBC driver, a timeout at the DB side, a restart of the database, the DB being run out of available connections, poor network quality, bad virusscanner/firewall/proxy, etc.
As it happens intermittely, a bug in the JDBC driver can be less or more excluded. Left behind the remaining possible causes. I suggest to start with looking in the logs of the DB server.
It's hard to say, but if I would check the actual version of the JDBC driver. Make sure it's 11.1.0.6.
Oracle doesn't include the database version in the filename. So the driver for 11.2 is the exact same name as the driver for 11.1 - ojdbc5.jar. I would extract the driver jar file, and find the MANIFEST.MF file, this will contain some version information. Make sure the version of the JDBC driver matches the version of your database. I suspect it may be a version issue, since there isn't a jar file named ojdbc14.jar on Oracle's 11.1.0.6 download page.
If the version matches - I'm out of ideas :)
Other thing that was causing me this problem was having the HOSTNAME settings wrong. My connection attempt was hanged at:
"main" prio=10 tid=0x00007f7cc8009000 nid=0x2f3a runnable [0x00007f7cce69e000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.getLocalHostName(Native Method)
at java.net.InetAddress.getLocalHost(InetAddress.java:1444)
at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:176)
at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:162)
at java.security.AccessController.doPrivileged(Native Method)
So make sure you have an entry for your hostname in /etc/hosts/.
If you issue a hostname command like this:
$ hostname
my.server.com
You need a line in your /etc/hosts:
127.0.0.1 my my.server.com
As per Bug https://bugs.openjdk.java.net/browse/JDK-6202721
Java will not consder -Djava.security.egd=file:/dev/urandom
It should be -Djava.security.egd=file:/dev/./urandom
Just to clarify - at least from what we found on our side!
It is an issue with the setup of the randomizer for Linux in the JDK distribution - and we found it in Java6, not sure about Java7.
The syntax for linux for the randomizer is file:///dev/urandom, but the entry in the file is (probably left/copied from Windows) as file:/dev/urandom.
So then Java probably falls back on the default, which happens to be /dev/random. And which doesn't work on a headless machine!!!
The root cause of this problem has to do with user authentication versions. For each database user, multiple password verifiers are kept in the database. Typically when you upgrade your database, a new password verifier will be added to the list, a stronger one. The following query shows the password verifier versions that are available for each user. For example:
SQL> SELECT PASSWORD_VERSIONS FROM DBA_USERS WHERE USERNAME='SCOTT';
PASSWORD_VERSIONS
-----------------
11G 12C
When upgrading to a newer driver you can use a newer version of the verifier because the driver and server negotiate the strongest possible verifier to to be used. This newer version of the verifier will be more secure and will involve generating larger random numbers or using more complex hashing functions which can explain why you see issues while establishing JDBC connections. As mentioned by other responses using /dev/urandom normally resolves these issues. You can also decide to downgrade your password verifier and make the newer driver use the same older password verifier that your previous driver was using. For example if you want to use the 10G password verifier (for testing purposes only), first you need to make sure it's available for your user.
Set SQLNET.ALLOWED_LOGON_VERSION_SERVER=8 in sqlnet.ora on the server. Then:
SQL> alter user scott identified by "tiger";
User altered.
SQL> SELECT PASSWORD_VERSIONS FROM DBA_USERS WHERE USERNAME='SCOTT';
PASSWORD_VERSIONS
-----------------
10G 11G 12C
Then you can force the JDBC thin driver to use the 10G verifier by setting this JDBC property oracle.jdbc.thinLogonCapability="o3". If you run into the error "ORA-28040: No matching authentication protocol" then that means your server is not allowing the 10G verifier to be used. If that's the case then you need to check your configuration again.
Note that the suggested solution of using /dev/urandom did work the first time for me but didn't work always after that.
DBA at my firm switched of 'SQL* net banners' and that fixed it permanently for me with or without the above.
I don't know what 'SQL* net banners' are, but am hoping by putting this information here that if you have(are) a DBA he(you) would know what to do.
Disabling SQL Net Banners saved us.
-Djava.security.egd=file:/dev/./urandom should be right! not -Djava.security.egd=file:/dev/../dev/urandom or -Djava.security.egd=file:///dev/urandom
I faced the same problem when a liquibase was executed from jenkins. Sporadically this error was thrown to the output and the liquibase change logs were not executed at all.
Solution provided: In the jenkin's maven project, the jdk was updated from jdk8-131 to any newer version (eg java8-162).
OracleXETNSListener - this service has to be started if it was disabled.
run -> services.msc
and look out for that services
Whenever I open a socket channel. If the client accepts then 1 file descriptor is created internally so I can create a maximum of 1024 clients in Linux.
But I want to create more clients without increasing file descriptor limit in Linux
(ulimit -n 20000)
So how can I create more sockets in Java?
If your session is limited to 1024 file descriptors you can't use more then that from a single JVM.
But since the ulimit is a per-process limitation, you could probably get around it by starting more JVMs (i.e. to get 2048 connections start two JVMs each using 1024).
If you are using UDP, can you multiplex on a single local socket youself? You'll be able to separate incoming packets by their source address and port.
If it's TCP you're out of luck, and the TIME_WAIT period after closing each socket will make things worse.
Why cant you increase the ulimit ? It seems like an artificial limitation. There is no way from java code (afaik) that allows you access to the system to reset the ulimit - it needs to be set before the process starts - in a startup script or something similar.
The JBoss startup scripts peform a 'ulimit -n $MAX_FD' before they start Jboss ...
Len
The limit RLIMIT_NOFILE is enforced by the operative system and limits the highest fd a process can create. One fd is used for every file, pipe and socket that is opened.
There are hard and soft limits. Any process (like your shell or jvm) is permitted to change the soft value but only a privileged process (like a shell run by the root user) can change the hard value .
a) If you are not permitted to change the limit on the machine, find someone that are.
b) If you for some reason can't be bothered to type ulimit, I guess you can call the underlying system call using JNA : man setrlimit(2). (.exec() won't do as it's a built in command)
See also Working With Ulimit
We recently upped our ulimit because our java process was throwing lots of "Too many files open" exceptions.
It is now 65536 and we have not had any issues.
If you really are looking at coping with a huge number of connections then the bast way to do it scalably would be to implement a lightweight dataserver process that has no responsibility other than accepting and forwarding data to a parent process.
That way as the each dataserver gets saturated you simply spawn a new instance to give yourself another 1024 connections. You could even have them exist on seperate machines if needed.
We have some applications that sometimes get into a bad state, but only in production (of course!). While taking a heap dump can help to gather state information, it's often easier to use a remote debugger. Setting this up is easy -- one need only add this to his command line:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=PORT
There seems to be no available security mechanism, so turning on debugging in production would effectively allow arbitrary code execution (via hotswap).
We have a mix of 1.4.2 and 1.5 Sun JVMs running on Solaris 9 and Linux (Redhat Enterprise 4). How can we enable secure debugging? Any other ways to achieve our goal of production server inspection?
Update: For JDK 1.5+ JVMs, one can specify an interface and port to which the debugger should bind. So, KarlP's suggestion of binding to loopback and just using a SSH tunnel to a local developer box should work given SSH is set up properly on the servers.
However, it seems that JDK1.4x does not allow an interface to be specified for the debug port. So, we can either block access to the debug port somewhere in the network or do some system-specific blocking in the OS itself (IPChains as Jared suggested, etc.)?
Update #2: This is a hack that will let us limit our risk, even on 1.4.2 JVMs:
Command line params:
-Xdebug
-Xrunjdwp:
transport=dt_socket,
server=y,
suspend=n,
address=9001,
onthrow=com.whatever.TurnOnDebuggerException,
launch=nothing
Java Code to turn on debugger:
try {
throw new TurnOnDebuggerException();
} catch (TurnOnDebugger td) {
//Nothing
}
TurnOnDebuggerException can be any exception guaranteed not to be thrown anywhere else.
I tested this on a Windows box to prove that (1) the debugger port does not receive connections initially, and (2) throwing the TurnOnDebugger exception as shown above causes the debugger to come alive. The launch parameter was required (at least on JDK1.4.2), but a garbage value was handled gracefully by the JVM.
We're planning on making a small servlet that, behind appropriate security, can allow us to turn on the debugger. Of course, one can't turn it off afterward, and the debugger still listens promiscuously once its on. But, these are limitations we're willing to accept as debugging of a production system will always result in a restart afterward.
Update #3: I ended up writing three classes: (1) TurnOnDebuggerException, a plain 'ol Java exception, (2) DebuggerPoller, a background thread the checks for the existence of a specified file on the filesystem, and (3) DebuggerMainWrapper, a class that kicks off the polling thread and then reflectively calls the main method of another specified class.
This is how its used:
Replace your "main" class with DebuggerMainWrapper in your start-up scripts
Add two system (-D) params, one specifying the real main class, and the other specifying a file on the filesystem.
Configure the debugger on the command line with the onthrow=com.whatever.TurnOnDebuggerException part added
Add a jar with the three classes mentioned above to the classpath.
Now, when you start up your JVM everything is the same except that a background poller thread is started. Presuming that the file (ours is called TurnOnDebugger) doesn't initially exist, the poller checks for it every N seconds. When the poller first notices it, it throws and immediately catches the TurnOnDebuggerException. Then, the agent is kicked off.
You can't turn it back off, and the machine is not terribly secure when its on. On the upside, I don't think the debugger allows for multiple simultaneous connections, so maintaining a debugging connection is your best defense. We chose the file notification method because it allowed us to piggyback off of our existing Unix authen/author by specifying the trigger file in a directory where only the proper uses have rights. You could easily build a little war file that achieved the same purpose via a socket connection. Of course, since we can't turn off the debugger, we'll only use it to gather data before killing off a sick application. If anyone wants this code, please let me know. However, it will only take you a few minutes to throw it together yourself.
If you use SSH you can allow tunneling and tunnel a port to your local host. No development required, all done using sshd, ssh and/or putty.
The debug socket on your java server can be set up on the local interface 127.0.0.1.
You're absolutely right: the Java Debugging API is inherently insecure. You can, however, limit it to UNIX domain sockets, and write a proxy with SSL/SSH to let you have authenticated and encrypted external connections that are then proxied into the UNIX domain socket. That at least reduces your exposure to someone who can get a process into the server, or someone who can crack your SSL.
Export information/services into JMX and then use RMI+SSL to access it remotely. Your situation is what JMX is designed for (the M stands for Management).
Good question.
I'm not aware of any built-in ability to encrypt connections to the debugging port.
There may be a much better/easier solution, but I would do the following:
Put the production machine behind a firewall that blocks access to the debugging port(s).
Run a proxy process on the host itself that connects to the port, and encrypts the input and output from the socket.
Run a proxy client on the debugging workstation that also encrypts/decrypts the input. Have this connect to the server proxy. Communication between them would be encrypted.
Connect your debugger to the proxy client.