I have a problem with MantaRay JMS: I use a static world map because autodiscovery wouldn't work in our network. If more than 10 peers are offline, I get an error 4226.
The problem is: Microsoft set a limit of 10 half-open connections with Windows XP SP2. MantaRay tries to contact every peer, and starts a lot of connections. The first 10 connections are Ok, then when the 11th starts, our software must wait for another connection to time out. Any other program trying to access the network on the same PC times out.
Strange thing is: on some PC the connection times out after 1-2 seconds, and the problem has almost no consequences, on some other, we have to wait 10 or 20 seconds. But according to Microsoft, there's no way to configure the default TCP connect timeout directly, and there are other factors (network switches, routers, VPN...) which could influence that.
I looked at the MantaRay source code, and tried to find a way to set the TCP connect timeout, but MantaRay uses SohetChannels instead of "regular" sockets, and the connect() method has no timeout. Am I missing something?
You could also patch the TCP/IP connection limit of WinXP... if you don't mind using such things. There are several sites offering patches. Just search Google for "change winxp tcp connection limit" and you'll find most of them. But use those tools on your own risk. Patching the code to work around that limit should be a better approach.
Problem solved.
I replaced the whole MantaRay with a much simpler JMS provider I wrote: I send a first test message over UDP, a peer is allowed to open a TCP connection only after this first message was received.
This taught me to be careful when using open-source (GPL) software.
Related
I've got a Java program that opens a TCP stream and connects to a listening port on a remote server. I send a request to the server and I receive a response. I then let the stream sit idle for 60 minutes. At that point if I write a new request it will not arrive at the server. In short order TCP/IP will let me know that the connection has gone away.
My client code is running on a Windows laptop which is connected to a corporate environment via a VPN router. The server is whirring away up in Canada, far away from me here in central Massachusetts USA. I'm likely being routed through multiple pieces of networking equipment. I have no idea which one is causing the stream to break. (I keep thinking of Ghostbusters and "Don't cross the streams!")
What is the best term to use when a piece of equipment specifically "forgets" about a TCP connection which has been idle, causing it to break? Is that half-open, half-closed, or just plain gone?
I want to be able to simulate this timeout scenario entirely within my home lab so that I can perform easier testing -- for example where I don't have to wait for 60 minutes! What's a good technique, and what is the appropriate equipment I should use to simulate this "disconnect"? I've got extra switches here at home, as well as one old (and fiesty!) WRT router that could use some lovin'.
I do not want to enable keepalive to mask the problem. Keepalive won't prevent all possible stream disconnection scenarios, AFAIK. I want to do the best that I can at letting the problem occur and handling it quickly and cleanly when it does.
Thank you very much,
Bill S
I am running an API on 10 different servers, all of them are behind a firewall. I am using jetty 8 to serve all the http requests. The use case for this API is short lived connections.
A few month ago I started to get random Too many open file descriptors errors. These errors make the server completely unresponsive and I need to restart the jetty server in order to fix that. Today this happened 0-10 times a day depending on the traffic I am getting.
After some investigations, I noticed that I am exhausting the number of available connections because all of them are stuck in the TIME_WAIT state so I can't create new ones.
ss -s
TCP: 13392 (estab 1549, closed 11439, orphaned 9, synrecv 0, timewait *11438*/0), ports 932
On this example the number of connections in TIME_WAIT state is pretty low but it can go up to 50k.
I have been trying several kernel tweaks and I also tried to set the SO_LINGER timer to 1 second for jetty sockets. All these changes helped reduce the frequency but I am still getting errors regularly.
Also worth mentioning, I am receiving around 3k requests/second on each server and the cpu usage is very low. The bottleneck to scale my traffic today is this connection issue.
Does anyone have an idea of what I can do to handle that correctly ?
'Too many open file descriptors' is probably caused by a resource leak in your application.
The TIME_WAIT state is caused by being the end that first sends a close, instead of the end that first receives the close. You might want to reconsider your application protocol so that it is the client which closes first. This is not too hard to arrange. It falls out free if you use client-side connection pooling for example.
These two conditions are not related. The TIME_WAIT state can only occur on a port whose socket has already been closed. It does not cause 'too many open file descriptors' problems.
I am having problems keeping a socket connection in my application alive for a long period. It seems to disconnect after a short while (every 2 minutes) especially if idle.
It happens that I am writing an application with a Chat feature and this requires I keep a socket connection Alive for the life-tme of my application. I have already tried tricks like pinging server every minute, but then this constitute a drain of (Blackberry) resources. Blackberry Push isnt an option (as i have other clients other than BB devices).
Please does anyone know how I can solve this or still suggest another alternative way of going about this?
[also i am interested in how instant messaging cross-mobile platform apps like WhatsApp, Kiks, etc manage to pull this off without excessively draining resources (at least my Resource Monitor App doesnt cry foul]
We found same behavior with socket connection on several BB OS versions. We tried to play with socket connection options - timeout supported, timeout value, etc. But we didn't find anything reliable and workable on all platforms consistent.
So we ended up with next configuration:
Devices with push - keep socket open with pinging after inactivity timeout switch to the push
Devices without push - keep socket with longest available ping
Case 2 is mush rare and as well we selected longest the longest ping interval that keeps connection open. It doesn't work 100%, we saw that on some devices it still drops the connection. But we carefully handle this case with reopening new one.
Hope this helps. Success!
Blackberry Push isnt an option (as i have other clients other than BB devices).
It's complicated, but popular apps definitely take advantage of BlackBerry push, as well as push functionality offered on all the other mobile platforms. It's part of supporting BlackBerry "for real" not just as an afterthought. The short answer for you is to use the BlackBerry push service.
If I connect with Java to MySQL on my localhost server, I access instantaneously.
But if I connect outside of the localhost, from a network PC (192.168.1.100), it is very slow (4-5 seconds).
And, if I connect from a public IP to my MY SQL server, it is also very slow (6 seconds or more).
The "why" is already been answered. It's just the network latency.
You're probably also interested in how to "fix" it. The answer is: use a connection pool. If you're running a Java webapplication, use the webserver-provided connection pooling facilities. To take Tomcat as an example, check this manual. If you're running a Java desktop application, use a decent connection pool implementation like c3p0 (tutorial here).
Because your computer needs time to send packets to external servers and they need time to send packets back. It's called network latency, and is not an issue with Java specifically, but a general network issue.
It will always take longer to make a connection across the network than to make the same connection locally. However, assuming you have a fairly typical local network, 4-5 seconds sounds a bit extreme. My guess (and it is just a guess) would be that the majority of the extra time is being consumed by network name resolution (i.e. DNS and/or netbios).
I would suggest that you try the connection using a numeric IP address, rather than a name.
Network latency plus connection creation time would be my guess. I don't know what else you have between the client machine and the MySQL server.
4 seconds on connecting could be a DNS problem and cannot be just a pure network latency.
Try starting MySQL server with "skip-name-resolve" parameter to skip resolving client's IP into hostname. Prior to that, make sure your grant tables are based on IPs and 'localhost' instead of symbolic names.
I have a typical client server communication - Client sends data to the server, server processes that, and returns data to the client. The problem is that the process operation can take quite some time - order of magnitude - minutes. There are a few approaches that could be used to solve this.
Establish a connection, and keep it alive, until the operation is finished and the client receives the response.
Establish connection, send data, close the connection. Now the processing takes place and once it is finished the server could establish a connection to the client to send the data.
Establish a connection, send data, close the connection. Processing takes place. client asks server, every n minutes/seconds if the operation is finished. If the processing is finished the client fetches the data.
I was wondering which approach would be the best way to use. Is there maybe some "de facto" standard for solving this problem? How "expensive" is opening a socket in Java? Solution 1. seems pretty nasty to me, but 2. and 3. could do. The problem with solution 2. is that the server needs to know on which port the client is listening, while solution 3. adds some network overhead.
is good enought
will not work at many situations, for example wne client is under firewall, NAT, and so on. Server usually accepts incoming connections from everywhere, desktops usualy not
better than 1 just because you will haven't problems when connection is lost
solutions 1+3 - make long waiting connections, with periodical sleep and reconnect after. I mean: connect to server, wait 30 sec for data, if no data received, sleep for 10 sec, loop.
Opening sockets is sometimes expensive, but not so expensive that your data processing.
I see an immediate problem with option 2. If the client is behind a firewall, he might very well be allowed to connect and do the request, but the server might be prevented to connect back to the cilent.
As you say, option 1 looks a bit nasty (not too nasty though, could work well), so among the options listed, I would go for option 3. Perhaps the server could estimate the time that's left of the processing, and hint the client, in each poll, of when it's about time to check back.