Whenever I open a socket channel. If the client accepts then 1 file descriptor is created internally so I can create a maximum of 1024 clients in Linux.
But I want to create more clients without increasing file descriptor limit in Linux
(ulimit -n 20000)
So how can I create more sockets in Java?
If your session is limited to 1024 file descriptors you can't use more then that from a single JVM.
But since the ulimit is a per-process limitation, you could probably get around it by starting more JVMs (i.e. to get 2048 connections start two JVMs each using 1024).
If you are using UDP, can you multiplex on a single local socket youself? You'll be able to separate incoming packets by their source address and port.
If it's TCP you're out of luck, and the TIME_WAIT period after closing each socket will make things worse.
Why cant you increase the ulimit ? It seems like an artificial limitation. There is no way from java code (afaik) that allows you access to the system to reset the ulimit - it needs to be set before the process starts - in a startup script or something similar.
The JBoss startup scripts peform a 'ulimit -n $MAX_FD' before they start Jboss ...
Len
The limit RLIMIT_NOFILE is enforced by the operative system and limits the highest fd a process can create. One fd is used for every file, pipe and socket that is opened.
There are hard and soft limits. Any process (like your shell or jvm) is permitted to change the soft value but only a privileged process (like a shell run by the root user) can change the hard value .
a) If you are not permitted to change the limit on the machine, find someone that are.
b) If you for some reason can't be bothered to type ulimit, I guess you can call the underlying system call using JNA : man setrlimit(2). (.exec() won't do as it's a built in command)
See also Working With Ulimit
We recently upped our ulimit because our java process was throwing lots of "Too many files open" exceptions.
It is now 65536 and we have not had any issues.
If you really are looking at coping with a huge number of connections then the bast way to do it scalably would be to implement a lightweight dataserver process that has no responsibility other than accepting and forwarding data to a parent process.
That way as the each dataserver gets saturated you simply spawn a new instance to give yourself another 1024 connections. You could even have them exist on seperate machines if needed.
Related
I have number of java process running on my machine. I need to track how many times each process is getting restarted.
For Example:
Let us consider two java process
Process 1 - Its restarted for 5 times.
Process 2 - Its restarted for 2 times.
I'm able to get the PID, java command of the running processes. But I could not able to differentiate once the process got restarted. Because the PID changed after the restart. Also I can't consider the java command because two instance of same application which has same command.
So what are the other ways to track the java process restart ?
You want your processes to keep the same identity after a restart. Ideally, you would have a parameter, system property or environment variable telling the process its identity.
As you state in the question, this identity cannot be passed on the command line. Thus, the process has to "find" its identity by acquiring an exclusive resource.
This resource could be a shared system implementing locks but it is probably to complex.
As exclusive resources, we have network sockets. So you could make your processes artificially opening a socket in the sole objective to make it acquire an identity.
You can use the code from https://stackoverflow.com/a/116113/2242270 in order to open a socket in a range. The identity of the process is then the port that could be opened.
I have a Java application which initially reads 3 lakhs of data from my MYSQL database.Then it calls an API using an ExecutorService with newFixedThreadPool size=20.
After getting the response from the API it is inserting the responses to my DB.It is working fine for first 2000 rows(nearby).After that I am getting an error like following.
SQLError-com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The
driver was unable to create a connection due to an inability to
establish the client portion of a socket.
This is usually caused by a limit on the number of sockets imposed by
the operating system. This limit is usually configurable.
For Unix-based platforms, see the manual page for the 'ulimit'
command. Kernel or system reconfiguration may also be required.
For Windows-based platforms, see Microsoft Knowledge Base Article
196271 (Q196271).
Anyone could help me to fix this issue?
I was able to fix this problem by increasing the # of sockets that can be opened in Windows:
From the Windows Start menu, run the regedit.exe application
In the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters directory, create a new DWORD value named MaxUserPort with a decimal value of 65,000 (the default is 5,000)
After closing the regedit application, restart your computer.
(See also Increasing the number of Windows sockets or ports, Maximum Socket Connections)
A note of caution: When an application is using more than 5,000 socket connections, that may be an indication that system resources are not being used in a sustainable way. It would be good to investigate the root cause for why so many sockets are being opened simultaneously.
i am currently testing a server with an automatic test client that simulates a large number of users. Both the server and the client are written in Java. The client opens a tcp/ip connection for every user. Both the server and client run on Ubuntu linux, client runs on 11.04 and server on 10.04.
The testing went good up till 27000 concurrently open connections, after that i decided to jump to 36000 (the servers and clients resources weren't really all that used up at 27000 so i decided to make a slightly bigger jump). When i tried running the test for 36k i got the following exception on the client side:
java.net.BindException: cannot assign requested address
As far as i know at 36k i should still have free ports since not much else is running on either machine and tcp limits the port number at 2^16 which is 65536. Now since it is linux i also set the number of open files for the user to 100k with ulimit -n 100000.
But i am still getting the same exception.
I'm wondering what else could be a possible cause for the mentioned exception, or does linux in some other way limit the number of outgoing connections ?
Thanks in advance,
Danijel
By default, Linux picks dynamically assigned ports from the range 32768..61000. The others are available for static assignment, if you bind to a specific port number. The range can be changed if you want more of the ports to be available for dynamic assignment, but just be careful that you do not include ports that are used for specific services that you need (e.g. 6000 for X11). Also you should not allow ports < 1024 to be dynamically assigned since they are privileged. To check or change the range:
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
# echo "16384 65535" > /proc/sys/net/ipv4/ip_local_port_range
Hi I have created a socket and client program using java NIO.
My server and client are on different computers ,Server have LINUX OS and CLIENT have WINDOWS OS. Whenever I have created 1024 sockets on client my client machines supports but in server I got too many files open error.
So How to open 15000 sockets without any error in server.
Or is there any other way to connect with 15000 clients at the same time?
Thanks
Bapi
Ok, questioning why he needs 15K sockets is a separate discussion.
The answer is that you are hitting the user's file descriptor limit.
Log with the user you will use in the listener and do $ulimit -n to see the current limit.
Most likely 1024.
Using root edit the file /etc/security/limits.conf
and set ->
{username} soft nofile 65536
{username} hard nofile 65536
65536 is just a suggestion, you will need to figure that out from your app.
Log off, log in again and recheck with ulimit -n, to see it worked.
Your are probably going to need more than 15 fds for all that. Monitor your app with lsof.
Like this:
$lsof -p {pid} <- lists all file descriptors
$lsof -p {pid} | wc -l <- count them
By the way, you might also hit the system wide fd limit, so you need to check it:
$cat /proc/sys/fs/file-max
To increase that one, add this line to the /etc/sysctl.conf
#Maximum number of open FDs
fs.file-max = 65535
Why do you need to have 15000 sockets on one machine? Anyway, look at ulimit -n
If you're going to have 15,000 clients talking to your server (and possibly 200,000 in the future according to your comments) then I suspect you're going to have scalability problems servicing those clients once they're connected (if they connect).
I think you may need to step back and look at how you can architect your application and/or deployment to successfully achieve these sort of numbers.
We have some applications that sometimes get into a bad state, but only in production (of course!). While taking a heap dump can help to gather state information, it's often easier to use a remote debugger. Setting this up is easy -- one need only add this to his command line:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=PORT
There seems to be no available security mechanism, so turning on debugging in production would effectively allow arbitrary code execution (via hotswap).
We have a mix of 1.4.2 and 1.5 Sun JVMs running on Solaris 9 and Linux (Redhat Enterprise 4). How can we enable secure debugging? Any other ways to achieve our goal of production server inspection?
Update: For JDK 1.5+ JVMs, one can specify an interface and port to which the debugger should bind. So, KarlP's suggestion of binding to loopback and just using a SSH tunnel to a local developer box should work given SSH is set up properly on the servers.
However, it seems that JDK1.4x does not allow an interface to be specified for the debug port. So, we can either block access to the debug port somewhere in the network or do some system-specific blocking in the OS itself (IPChains as Jared suggested, etc.)?
Update #2: This is a hack that will let us limit our risk, even on 1.4.2 JVMs:
Command line params:
-Xdebug
-Xrunjdwp:
transport=dt_socket,
server=y,
suspend=n,
address=9001,
onthrow=com.whatever.TurnOnDebuggerException,
launch=nothing
Java Code to turn on debugger:
try {
throw new TurnOnDebuggerException();
} catch (TurnOnDebugger td) {
//Nothing
}
TurnOnDebuggerException can be any exception guaranteed not to be thrown anywhere else.
I tested this on a Windows box to prove that (1) the debugger port does not receive connections initially, and (2) throwing the TurnOnDebugger exception as shown above causes the debugger to come alive. The launch parameter was required (at least on JDK1.4.2), but a garbage value was handled gracefully by the JVM.
We're planning on making a small servlet that, behind appropriate security, can allow us to turn on the debugger. Of course, one can't turn it off afterward, and the debugger still listens promiscuously once its on. But, these are limitations we're willing to accept as debugging of a production system will always result in a restart afterward.
Update #3: I ended up writing three classes: (1) TurnOnDebuggerException, a plain 'ol Java exception, (2) DebuggerPoller, a background thread the checks for the existence of a specified file on the filesystem, and (3) DebuggerMainWrapper, a class that kicks off the polling thread and then reflectively calls the main method of another specified class.
This is how its used:
Replace your "main" class with DebuggerMainWrapper in your start-up scripts
Add two system (-D) params, one specifying the real main class, and the other specifying a file on the filesystem.
Configure the debugger on the command line with the onthrow=com.whatever.TurnOnDebuggerException part added
Add a jar with the three classes mentioned above to the classpath.
Now, when you start up your JVM everything is the same except that a background poller thread is started. Presuming that the file (ours is called TurnOnDebugger) doesn't initially exist, the poller checks for it every N seconds. When the poller first notices it, it throws and immediately catches the TurnOnDebuggerException. Then, the agent is kicked off.
You can't turn it back off, and the machine is not terribly secure when its on. On the upside, I don't think the debugger allows for multiple simultaneous connections, so maintaining a debugging connection is your best defense. We chose the file notification method because it allowed us to piggyback off of our existing Unix authen/author by specifying the trigger file in a directory where only the proper uses have rights. You could easily build a little war file that achieved the same purpose via a socket connection. Of course, since we can't turn off the debugger, we'll only use it to gather data before killing off a sick application. If anyone wants this code, please let me know. However, it will only take you a few minutes to throw it together yourself.
If you use SSH you can allow tunneling and tunnel a port to your local host. No development required, all done using sshd, ssh and/or putty.
The debug socket on your java server can be set up on the local interface 127.0.0.1.
You're absolutely right: the Java Debugging API is inherently insecure. You can, however, limit it to UNIX domain sockets, and write a proxy with SSL/SSH to let you have authenticated and encrypted external connections that are then proxied into the UNIX domain socket. That at least reduces your exposure to someone who can get a process into the server, or someone who can crack your SSL.
Export information/services into JMX and then use RMI+SSL to access it remotely. Your situation is what JMX is designed for (the M stands for Management).
Good question.
I'm not aware of any built-in ability to encrypt connections to the debugging port.
There may be a much better/easier solution, but I would do the following:
Put the production machine behind a firewall that blocks access to the debugging port(s).
Run a proxy process on the host itself that connects to the port, and encrypts the input and output from the socket.
Run a proxy client on the debugging workstation that also encrypts/decrypts the input. Have this connect to the server proxy. Communication between them would be encrypted.
Connect your debugger to the proxy client.