We Keep Coding

How prevent too many file open from close_wait connections

My program is fetching some images on a min.io server via their Java SDK.
The issue is that even after inputStream.close() the connections remain open from the java code. I can see it with lsof -p <PID>.
After a while, it disappears but sometimes it does not, I guess fast enough, and the java server throws some too many open files errors.
Is there like a garbage collector that removes the connections from the operating system?
How can I prevent these too many open files errors?
Just in case, here is the code:
public static byte[] getImageByImageBinaryId(String imagId) throws IOException {
InputStream object = null;
try {
object = getMinioClientClient().getObject(ServerProperties.MINIO_BUCKET_NAME, imagId);
return IOUtils.toByteArray(object);
} catch (Exception e) {
log.error(e);
} finally {
IOUtils.closeQuietly(object);
}
return null;
}

Internally minio-java uses OkHttp to make HTTP calls. OkHttp, like many Http clients, internally uses a connection pool to speed up repeated calls to the same location. If you need for connections to not persist you can pass in your own OkHttp client to one of the available constructors with your own pooling config but I do not recommend it.
Minio should probably expose a close method to clean up these resources but their expected use case probably involves clients living the whole life of your application.

Java threaded socket connection timeouts

I have to make simultaneous tcp socket connections every x seconds to multiple machines, in order to get something like a status update packet.
I use a Callable thread class, which creates a future task that connects to each machine, sends a query packet, and receives a reply which is returned to the main thread that creates all the callable objects.
My socket connection class is :
public class ClientConnect implements Callable<String> {
Connection con = null;
Statement st = null;
ResultSet rs = null;
String hostipp, hostnamee;
ClientConnect(String hostname, String hostip) {
hostnamee=hostname;
hostipp = hostip;
}
#Override
public String call() throws Exception {
return GetData();
}
private String GetData() {
Socket so = new Socket();
SocketAddress sa = null;
PrintWriter out = null;
BufferedReader in = null;
try {
sa = new InetSocketAddress(InetAddress.getByName(hostipp), 2223);
} catch (UnknownHostException e1) {
e1.printStackTrace();
}
try {
so.connect(sa, 10000);
out = new PrintWriter(so.getOutputStream(), true);
out.println("\1IDC_UPDATE\1");
in = new BufferedReader(new InputStreamReader(so.getInputStream()));
String [] response = in.readLine().split("\1");
out.close();in.close();so.close(); so = null;
try{
Integer.parseInt(response[2]);
} catch(NumberFormatException e) {
System.out.println("Number format exception");
return hostnamee + "|-1" ;
}
return hostnamee + "|" + response[2];
} catch (IOException e) {
try {
if(out!=null)out.close();
if(in!=null)in.close();
so.close();so = null;
return hostnamee + "|-1" ;
} catch (IOException e1) {
// TODO Auto-generated catch block
return hostnamee + "|-1" ;
}
}
}
}
And this is the way i create a pool of threads in my main class :
private void StartThreadPool()
{
ExecutorService pool = Executors.newFixedThreadPool(30);
List<Future<String>> list = new ArrayList<Future<String>>();
for (Map.Entry<String, String> entry : pc_nameip.entrySet())
{
Callable<String> worker = new ClientConnect(entry.getKey(),entry.getValue());
Future<String> submit = pool.submit(worker);
list.add(submit);
}
for (Future<String> future : list) {
try {
String threadresult;
threadresult = future.get();
//........ PROCESS DATA HERE!..........//
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
}
The pc_nameip map contains (hostname, hostip) values and for every entry i create a ClientConnect thread object.
My problem is that when my list of machines contains lets say 10 pcs (which most of them are not alive), i get a lot of timeout exceptions (in alive pcs) even though my timeout limit is set to 10 seconds.
If i force the list to contain a single working pc, I have no problem.
The timeouts are pretty random, no clue what's causing them.
All machines are in a local network, the remote servers are written by my also (in C/C++) and been working in another setup for more than 2 years without any problems.
Am i missing something or could it be an os network restriction problem?
I am testing this code on windows xp sp3. Thanks in advance!
UPDATE:
After creating two new server machines, and keeping one that was getting a lot of timeouts, i have the following results :
For 100 thread runs over 20 minutes :
NEW_SERVER1 : 99 successful connections/ 1 timeouts
NEW_SERVER2 : 94 successful connections/ 6 timeouts
OLD_SERVER : 57 successful connections/ 43 timeouts
Other info :
- I experienced a JRE crash (EXCEPTION_ACCESS_VIOLATION (0xc0000005)) once and had to restart the application.
- I noticed that while the app was running my network connection was struggling as i was browsing the internet. I have no idea if this is expected but i think my having at MAX 15 threads is not that much.
So, fisrt of all my old servers had some kind of problem. No idea what that was, since my new servers were created from the same OS image.
Secondly, although the timeout percentage has dropped dramatically, i still think it is uncommon to get even one timeout in a small LAN like ours. But this could be a server's application part problem.
Finally my point of view is that, apart from the old server's problem (i still cannot beleive i lost so much time with that!), there must be either a server app bug, or a JDK related bug (since i experienced that JRE crash).
p.s. I use Eclipse as IDE and my JRE is the latest.
If any of the above ring any bells to you, please comment.
Thank you.
-----EDIT-----
Could it be that PrintWriter and/or BufferedReader are not actually thread safe????!!!?
----NEW EDIT 09 Sep 2013----
After re-reading all the comments and thanks to #Gray and his comment :
When you run multiple servers does the first couple work and the rest of them timeout? Might be interesting to put a small sleep in your fork loop (like 10 or 100ms) to see if it works that way.
I rearanged the tree list of the hosts/ip's and got some really strange results.
It seems that if an alive host is placed on top of the tree list, thus being first to start a socket connection, has no problem connecting and receiving packets without any delay or timeout.
On the contrary, if an alive host is placed at the bottom of the list, with several dead hosts before it, it just takes too long to connect and with my previous timeout of 10 secs it failed to connect. But after changing the timeout to 60 seconds (thanks to #EJP) i realised that no timeouts are occuring!
It just takes too long to connect (more than 20 seconds in some occasions).
Something is blobking new socket connections, and it isn't that the hosts or network is to busy to respond.
I have some debug data here, if you would like to take a look :
http://pastebin.com/2m8jDwKL

You could simply check for availability before you connect to the socket. There is an answer who provides some kind of hackish workaround https://stackoverflow.com/a/10145643/1809463
Process p1 = java.lang.Runtime.getRuntime().exec("ping -c 1 " + ip);
int returnVal = p1.waitFor();
boolean reachable = (returnVal==0);
by jayunit100
It should work on unix and windows, since ping is a common program.

My problem is that when my list of machines contains lets say 10 pcs (which most of them are not alive), i get a lot of timeout exceptions (in alive pcs) even though my timeout limit is set to 10 seconds.
So as I understand the problem, if you have (for example) 10 PCs in your map and 1 is alive and the other 9 are not online, all 10 connections time out. If you just put the 1 alive PC in the map, it shows up as fine.
This points to some sort of concurrency problem but I can't see it. I would have thought that there was some sort of shared data that was not being locked or something. I see your test code is using Statement and ResultSet. Maybe there is a database connection that is being shared without locking or something? Can you try just returning the result string and printing it out?
Less likely is some sort of network or firewall configuration but the idea that one failed connection would cause another to fail is just strange. Maybe try running your program on one of the servers or from another computer?
If I try your test code, it seems to work fine. Here's the source code for my test class. It has no problems contacting a combination of online and offline hosts.
Lastly some quick comments about your code:
You should close the streams, readers, and sockets in a finally block. Check my test class for a better pattern there.
You should return a small Result class instead of passing back a String that they has to be parsed.
Hope this helps.

After a lot of reading and experimentation i will have to answer my own question (if i am allowed to do of course).
Java just can't handle concurrent multiple socket connections without adding a big performance overhead. At least in a Core2Duo/4GB RAM/ Windows XP machine.
Creating multiple concurrent socket connections to remote hosts (using of course the code i posted) creates some kind of resource bottleneck, or blocking situation, wich i am still not aware of.
If you try to connect to 20 hosts simultaneously, and a lot of them are disconnected, then you cannot guarantee a "fast" connection to the alive ones.
You will get connected but could be after 20-25 seconds. Meaning that you'll have to set socket timeout to something like 60 seconds. (not acceptable for my application)
If an alive host is lucky to start its connection try first (having in mind that concurrency is not absolute. the for loop still has sequentiality), then he will probably get connected very fast and get a response.
If it is unlucky, the socket.connect() method will block for some time, depending on how many are the hosts before it that will timeout eventually.
After adding a small sleep between the pool.submit(worker) method calls (100 ms) i realised that it makes some difference. I get to connect faster to the "unlucky" hosts. But still if the list of dead hosts is increased, the results are almost the same.
If i edit my host list and place a previously "unlucky" host at the top (before dead hosts), all problems dissapear...
So, for some reason the socket.connect() method creates a form of bottleneck when the hosts to connect to are many, and not alive. Be it a JVM problem, a OS limitation or bad coding from my side, i have no clue...
I will try a different coding approach and hopefully tommorow i will post some feedback.
p.s. This answer made me think of my problem :
https://stackoverflow.com/a/4351360/2025271

Java Login Server Flood Protection

my java server is currently being DDoSed by thousands of different ip's. To counter it, i've been blocking the ip's through the dedicated servers firewall. However just more and more keep coming. This attack is flooding my login server causing players to not be able to login.
I've asked on other places such as a forum for my type of java application, and i've gotten barely any answers. However they told me it's a form of DDoS, also another user referred me to this however I don't believe this would work. http://www.java2s.com/Open-Source/Java-Document/Net/mina-2.0.0-M1/org/apache/mina/filter/firewall/ConnectionThrottleFilter.java.htm
So apparently it's a DoS on my login server, anyone have any clues? I'll provide the login server's main framework below.
try {
shutdownClientHandler = false;
clientListener = new java.net.ServerSocket(serverlistenerPort, 1, null);
misc.println("Login server is now online!");
while (true) {
try {
java.net.Socket s = clientListener.accept();
s.setTcpNoDelay(true);
String connectingHost = s.getInetAddress().getHostName();
if (true) {
if (checkLog("flooders", connectingHost)) {
//misc.println("Flooder Detected, closing connection.");
s.close(); // Closes the connection immediately.
} else {
connections.add(connectingHost);
if (checkHost(connectingHost)) {
misc.println("Connection from "+connectingHost+":"+s.getPort());
try {
playerHandler.newPlayerClient(s, connectingHost);
} catch (Exception E) {misc.println("Login Server crash detected. Protecting...");
s.close();
}
} else {
misc.println("ClientHandler: Rejected "+connectingHost+":"+s.getPort());
s.close();
}
}
} else {
misc.println("ClientHandler: Rejected "+connectingHost+":"+s.getPort());
s.close();
}
Thread.sleep(30;
} catch (Exception e) {
logError(e.getMessage());
}
}
} catch (java.io.IOException ioe) {
if (!shutdownClientHandler) {
misc.println("World is already online!");
} else {
misc.println("ClientHandler was shut down.");
}
}

There are a few options to try to deal with a DDoS, but you need to make certain certain parts are optimized.
For example, your checkLog function needs to be amazingly fast, so you may want to use a 'hashmap' perhaps, so you can quickly check, but, to ensure everything goes fast you may want to look at the ConcurrentHashMap (http://download.oracle.com/javase/1,5.0/docs/api/java/util/concurrent/ConcurrentHashMap.html), so you can better read and write to it.
You could also look at load-balancing your front-end, so that one server isn't having to do everything.
Depending on what actions needs to be done on the server, find a quick way to verify that someone has been authenticated, but this could slow down your server if you don't have special hardware to better handle decryption of a token, for example.
You may also want to look at using NIO (http://tutorials.jenkov.com/java-nio/index.html), to better scale to a large number of connections.

Ideally, you would not want to have the malicious requests hit your application at all but rather intercept/drop the traffic before it reaches your application servers. This way your read clients will not encounter any issues e. g. with degraded performance while the attack is under way.
ServerFault may be a good place to ask, if you intend going down that road.

How can i check if MySQL and Tomcat are running?

I've created a Java application that is split in different subcomponents, each of those runs on a separate Tomcat instance. Also, some components use a MySQL db through Hibernate.
I'm now creating an administration console where it's reported the status of all my Tomcat instances and of MySQL. I don't need detailed information, but knowing if they are running or not it's enough.
What could be the best solution to do that?
Thanks

Most straightforward way would be to just connect the server and see if it succeeds.
MySQL:
Connection connection = null;
try {
connection = DriverManager.getConnection(url, username, password);
// Succes!
} catch (SQLException e) {
// Fail!
} finally {
if (connection != null) try { connection.close(); } catch (SQLException ignore) {}
}
Tomcat:
try {
new URL(url).openConnection().connect();
// Succes!
} catch (IOException e) {
// Fail!
}
If you want a bit more specific status, e.g. checking if a certain DB table is available or a specific webapp resource is available, then you have to fire a more specific SELECT statement or HTTP request respectively.

I assume that you know the ports of which are running in advance (or from configuration files). The easiest way to check is to make socket connections to those ports like a telnet program does. Something like:
public boolean isServerUp(int port) {
boolean isUp = false;
try {
Socket socket = new Socket("127.0.0.1", port);
// Server is up
isUp = true;
socket.close();
}
catch (IOException e)
{
// Server is down
}
return isUp;
}
Usage:
isTomcatUp = isServerUp(8080);
isMysqlUp = isServerUp(3306);
However, I would say that is a false-negative check.. Sometimes it says server UP but the server is stuck or not responding...

I would make sure that what ever monitoring you setup is actually exercising some code. Monitoring the JVM via jmx can also be helpful after the fact. Check out http://www.cacti.net/ .

Firing a simple fixed query through MySQL
SELECT 'a-ok';
and have the .jsp return that a-ok text. If it times out and/or doesn't respond with a-ok, then something's hinky. If you need something more detailed, you can add extra checks, like requesting now() or something bigger, like SHOW INNODB STATUS.

The easiest thing is to look for the MySQL and Tomcat PID files. You need to look at your start scripts to make sure of the exact location, but once you find it, you simply test for existence of the pid file.

Create a servlet as a status page. In the servlet perform a cheap query, if the query succeeds let the servlet print OK otherwise Error. Put the servlet into a war and deploy it to all instances.
This could be used for checks in yor admin console by doing a loop over all instances.

I'd create a simple REST webservice that runs on each Tomcat instance and does a no-op query against the database. That makes it easy to drive from anywhere (command line, web app, GUI app, etc.)
If these are publicly available servers you can use a service like binarycanary.com to poll a page or service in your app.

How to detect internet connectivity using java program

How to write a java program which will tell me whether I have internet access or not. I donot want to ping or create connection with some external url because if that server will be down then my program will not work. I want reliable way to detect which will tell me 100% guarantee that whether I have internet connection or not irrespective of my Operating System. I want the program for the computers who are directly connected to internet.
I have tried with the below program
URL url = new URL("http://www.xyz.com/");
URLConnection conn = url.openConnection();
conn.connect();
I want something more appropriate than this program
Thanks
Sunil Kumar Sahoo

It depends on what you mean by "internet" connection. Many computers are not connected directly to the internet, so even if you could check whether they have a network connection, it doesn't always mean they can access the internet.
The only 100% reliable way to test whether the computer can access some other server is to actually try.

Effective connectivity to the internet (i.e. where you can actually do stuff) depends on lots of things being correct, on your machine, your local net, your router, your modem, your ISP and so on. There are lots of places where a failure or misconfiguration will partly or completely block network access.
It is impossible to test all of these potential failure points with any certainty ... or even to enumerate them. (For example, you typically have no way of knowing what is happening inside your ISP's networking infrastructure.)
As #codeka says: "the only 100% reliable way to test whether the computer can access some other server is to actually try".

I think if you were to open up a HTTP session with all of:
www.google.com
www.microsoft.com
www.ibm.com
www.ford.com
and at least one of them came back with a valid response, you would have internet connectivity. Don't keep testing once you get a valid response since that would be a waste.
Feel free to expand on that list with some more mega-corporations in case you fear that all four of them may be down at the same time :-)
Of course, even that method can be tricked if someone has taken control of your DNS servers but it's probably about as reliable as you're going to get.

Just put a try/catch block around the code you mentioned. If an exception is thrown/caught then you don't have connectivity.
boolean connectivity;
try {
URL url = new URL("http://www.xyz.com/");
URLConnection conn = url.openConnection();
conn.connect();
connectivity = true;
} catch (Exception e) {
connectivity = false;
}
For better results investigate what kind of exceptions can be thrown and handle each individually.

You can check the connectivity by ask the Internet Protocol from InetAddress class. If you get an exception, or for example you use getLocalHost() -- which is returns the address of the local host -- give you the following output:
localhost/127.0.0.1 instead of fully qualified name like for example jaf-stephen-lenovoG40-80/152.6.44.13 then you're not connected to the Internet.
public static void main(String[] args) {
try {
InetAddress address = InetAddress.getByName("www.facebook.com");
System.out.println(address);
} catch (UnknownHostException e) {
System.err.println("Couldn't find www.facebook.com");
}
}
if you're connected to the Internet, you'll get the following output:
www.facebook.com/31.13.78.35

Enumaration<NetworkInterface> networkInterface = null;
networkInterface = NetworkInterface.getNetworkInterfaces();
for(NetworkInterface interface : Collections.list(networkInterface)){
System.out.println("Internet Available status is :"+ interface.isUp());
}