My program is fetching some images on a min.io server via their Java SDK.
The issue is that even after inputStream.close() the connections remain open from the java code. I can see it with lsof -p <PID>.
After a while, it disappears but sometimes it does not, I guess fast enough, and the java server throws some too many open files errors.
Is there like a garbage collector that removes the connections from the operating system?
How can I prevent these too many open files errors?
Just in case, here is the code:
public static byte[] getImageByImageBinaryId(String imagId) throws IOException {
InputStream object = null;
try {
object = getMinioClientClient().getObject(ServerProperties.MINIO_BUCKET_NAME, imagId);
return IOUtils.toByteArray(object);
} catch (Exception e) {
log.error(e);
} finally {
IOUtils.closeQuietly(object);
}
return null;
}
Internally minio-java uses OkHttp to make HTTP calls. OkHttp, like many Http clients, internally uses a connection pool to speed up repeated calls to the same location. If you need for connections to not persist you can pass in your own OkHttp client to one of the available constructors with your own pooling config but I do not recommend it.
Minio should probably expose a close method to clean up these resources but their expected use case probably involves clients living the whole life of your application.
Related
I have an application that does a lot work on S3, mostly downloading files from it. I am seeing a lot of these kind of errors and I'd like to know if this is something on my code or if the service is really unreliable like this.
The code I'm using to read from the S3 object stream is as follows:
public static final void write(InputStream stream, OutputStream output) {
byte[] buffer = new byte[1024];
int read = -1;
try {
while ((read = stream.read(buffer)) != -1) {
output.write(buffer, 0, read);
}
stream.close();
output.flush();
output.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
This OutputStream is a new BufferedOutputStream( new FileOutputStream( file ) ). I am using the latest version of the Amazon S3 Java client and this call is retried four times before giving up. So, after trying this for 4 times it still fails.
Any hints or tips on how I could possibly improve this are appreciated.
I just managed to overcome a very similar problem. In my case the exception I was getting was identical; it happened for larger files but not for small files, and it never happened at all while stepping through the debugger.
The root cause of the problem was that the AmazonS3Client object was getting garbage collected in the middle of the download, which caused the network connection to break. This happened because I was constructing a new AmazonS3Client object with every call to load a file, while the preferred use case is to create a long-lasting client object that survives across calls - or at least is guaranteed to be around during the entirety of the download. So, the simple remedy is to make sure a reference to the AmazonS3Client is kept around so that it doesn't get GC'd.
A link on the AWS forums that helped me is here: https://forums.aws.amazon.com/thread.jspa?threadID=83326
The network is closing the connection, prior to the client getting all the data, for one reason or another, that's what is going on.
Part of any HTTP Request is the content length, Your code is getting the header, saying hey buddy, here's data, and its this much of it.. and then the connection is dropping before the client has read all of the data.. so its bombing out with the exception.
I'd look at your OS/NETWORK/JVM connection timeout settings (though JVM generally inherit from the OS in this situation). The key is to figure out what part of the network is causing the problem. Is it your computer level settings saying, nope not going to wait any longer for packets.. is it that you are using a non blocking read, which has a timeout setting in your code, where it is saying, hey, haven't gotten any data from the server since longer than I'm supposed to wait so I'm going to drop the connection and exception. etc etc etc.
Best bet is to low level snoop the packet traffic and trace backwards, to see where the connection drop is happening, or see if you can up timeouts in things you can control, like your software, and OS/JVM.
First of all, your code is operating entirely normally if (and only if) you suffer connectivity troubles between yourself and Amazon S3. As Michael Slade points out, standard connection-level debugging advice applies.
As to your actual source code, I note a few code smells you should be aware of. Annotating them directly in the source:
public static final void write(InputStream stream, OutputStream output) {
byte[] buffer = new byte[1024]; // !! Abstract 1024 into a constant to make
// this easier to configure and understand.
int read = -1;
try {
while ((read = stream.read(buffer)) != -1) {
output.write(buffer, 0, read);
}
stream.close(); // !! Unexpected side effects: closing of your passed in
// InputStream. This may have unexpected results if your
// stream type supports reset, and currently carries no
// visible documentation.
output.flush(); // !! Violation of RAII. Refactor this into a finally block,
output.close(); // a la Reference 1 (below).
} catch (IOException e) {
throw new RuntimeException(e); // !! Possibly indicative of an outer
// try-catch block for RuntimeException.
// Consider keeping this as IOException.
}
}
(Reference 1)
Otherwise, the code itself seems fine. IO exceptions should be expected occurrences in situations where you're connecting to a fickle remote host, and your best course of action is to draft a sane policy to cache and reconnect in these scenarios.
Try using wireshark to see what is happening on the wire when this happens.
Try temporarily replacing S3 with your own web server and see if the problem persists. If it does it's your code and not S3.
The fact that it's random suggests network issues between your host and some of the S3 hosts.
Also S3 could close slow connections according to my experience.
I would take a very close look at the network equipment nearest your client app. This problem smacks of some network device dropping packets between you and the service. Look to see if there was a starting point when the problem first occurred. Was there any change like a firmware update to a router or replacement of a switch around that time?
Verify your bandwidth usage against the amount purchased from your ISP. Are there times of the day where you're approaching that limit? Can you obtain graphs of your bandwidth usage? See if the premature terminations can be correlated with high-bandwidth usage, particularly if it approaches some known limit. Does the problem seem to pick on smaller files and on large files only when they're almost finished downloading? Purchasing more bandwidth from your ISP may fix the problem.
i have a java project, works as a server. when an instance of this project running, i can run another instance.
how can i avoid running of more than one instance on same java project at the same time?
(Stop the server when another instance is detected)
import java.net.ServerSocket;
.....
private static final int PORT = 9999;
private static ServerSocket socket;
public static void main(String[] args) {
try {
socket = new ServerSocket(PORT, 0, InetAddress.getByAddress(new byte[] { 127, 0, 0, 1 }));
{/*here write your own code taht must be run in the main*/}
} catch (BindException e) {
System.err.println("**********************************Already running.");
System.exit(1);
} catch (IOException e) {
System.err.println("************************************Unexpected error.");
e.printStackTrace();
System.exit(2);
} catch (Exception e) {
System.err.println("************************************ Error");
System.exit(3);
}
}
i used this code and it work try it
Easiest way is to use lock file, this causes problems if the app crashed. Try writing the pid into the lock file, you can check if that pid exists (although not natively maybe in a wrapper shell script).
If you are running server can you not check if a port is open, or better still maybe a jmx instance on a known port.
I totally support #vickirk - his approach allows the second "un-needed" instance of your server become "dormant" instead of simply terminating, i.e. periodically run to perform a check if the "active" instance is still actually active/present, and take over if it went down.
In the distrubuted case, if the requirement is to have a single server instance spanning multiple machines, the approach is still to find a common resource that can be locked, physically or logically. For that purpose, I personally use a control database table where an active process writes its PID and "heartbeat", and all others are checking for that "heartbeat" to be fairly recent, and become active if its not.
you can write simple command line script for app start - that check is server runs before actually run new instance. Just check url with wget for example...
I've created an MMO for the Android phone and use a Java server with TCP/IP sockets. Everything generally works fine, but after about a day of clients logging on and off my network becomes extremely laggy -- even if there aren't clients connected. NETSTAT shows no lingering connections, but there is obviously something terribly wrong going on.
If I do a full reboot everything magically is fine again, but this isn't a tenable solution for the long-term. This is what my disconnect method looks like (on both ends):
public final void disconnect()
{
Alive = false;
Log.write("Disconnecting " + _socket.getRemoteSocketAddress());
try
{
_socket.shutdownInput();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_socket.shutdownOutput();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_input.close();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_output.close();
}
catch (final Exception e)
{
Log.write(e);
}
try
{
_socket.close();
}
catch (final Exception e)
{
Log.write(e);
}
}
_input and _output are BufferedInputStream and BufferedOutputStream spawned from the socket. According to documentation calling shutdownInput() and shutdownOutput() shouldn't be necessary, but I'm throwing everything I possibly can at this.
I instantiate the sockets with default settings -- I'm not touching soLinger, KeepAlive, noDelay or anything like that. I do not have any timeouts set on send/receive. I've tried using WireShark but it reveals nothing unusual, just like NETSTAT.
I'm pretty desperate for answers on this. I've put a lot of effort into this project and am frustrated with what appears to be a serious hidden flaw in Java's default TCP implementation.
Get rid of shutdownInput() and shutdownOutput() and all the closes except the close for the BufferedOutputStream, and a subsequent close on the socket itself in a finally block as a belt & braces. You are shutting down and closing everything else before the output stream, which prevents it from flushing. Closing the output stream flushes it and closes the socket. That's all you need.
OP here, unable to comment on original post.
Restarting the server process does not appear to resolve the issue. The network remains very "laggy" even several minutes after shutting down the server entirely.
By "laggy" I mean the connection becomes extremely slow with both up and down traffic. Trying to load websites, or upload to my FTP, is painfully slow like I'm on a 14.4k modem (I'm on a 15mbs fiber). Internet Speed Tests don't even work when it is in this state -- I get an error about not finding the file, when the websites eventually load up.
All of this instantly clears up after a reboot, and only after a reboot.
I modified my disconnect method as EJP suggested, but the problem persists.
Server runs on a Windows 7 installation, latest version of Java / Java SDK. The server has 16gb of RAM, although it's possible I'm not allocating it properly for the JVM to use fully. No stray threads or processes appear to be present. I'll see what JVISUALVM says. – jysend 13 mins ago
Nothing unusual in JVISUALVM -- 10mb heap, 50% CPU use, 3160 objects (expected), 27 live threads out of 437 started. Server has been running for about 18 hours; loading up CNN's front page takes about a minute, and the normal speed test I use (first hit googling Speed Test) won't even load the page. NETSTAT shows no lingering connections. Ran all up to date antivirus. Server has run 24/7 in the past without any issues -- it is only when I started running this Java server on it that this started to happen.
I've created a Java application that is split in different subcomponents, each of those runs on a separate Tomcat instance. Also, some components use a MySQL db through Hibernate.
I'm now creating an administration console where it's reported the status of all my Tomcat instances and of MySQL. I don't need detailed information, but knowing if they are running or not it's enough.
What could be the best solution to do that?
Thanks
Most straightforward way would be to just connect the server and see if it succeeds.
MySQL:
Connection connection = null;
try {
connection = DriverManager.getConnection(url, username, password);
// Succes!
} catch (SQLException e) {
// Fail!
} finally {
if (connection != null) try { connection.close(); } catch (SQLException ignore) {}
}
Tomcat:
try {
new URL(url).openConnection().connect();
// Succes!
} catch (IOException e) {
// Fail!
}
If you want a bit more specific status, e.g. checking if a certain DB table is available or a specific webapp resource is available, then you have to fire a more specific SELECT statement or HTTP request respectively.
I assume that you know the ports of which are running in advance (or from configuration files). The easiest way to check is to make socket connections to those ports like a telnet program does. Something like:
public boolean isServerUp(int port) {
boolean isUp = false;
try {
Socket socket = new Socket("127.0.0.1", port);
// Server is up
isUp = true;
socket.close();
}
catch (IOException e)
{
// Server is down
}
return isUp;
}
Usage:
isTomcatUp = isServerUp(8080);
isMysqlUp = isServerUp(3306);
However, I would say that is a false-negative check.. Sometimes it says server UP but the server is stuck or not responding...
I would make sure that what ever monitoring you setup is actually exercising some code. Monitoring the JVM via jmx can also be helpful after the fact. Check out http://www.cacti.net/ .
Firing a simple fixed query through MySQL
SELECT 'a-ok';
and have the .jsp return that a-ok text. If it times out and/or doesn't respond with a-ok, then something's hinky. If you need something more detailed, you can add extra checks, like requesting now() or something bigger, like SHOW INNODB STATUS.
The easiest thing is to look for the MySQL and Tomcat PID files. You need to look at your start scripts to make sure of the exact location, but once you find it, you simply test for existence of the pid file.
Create a servlet as a status page. In the servlet perform a cheap query, if the query succeeds let the servlet print OK otherwise Error. Put the servlet into a war and deploy it to all instances.
This could be used for checks in yor admin console by doing a loop over all instances.
I'd create a simple REST webservice that runs on each Tomcat instance and does a no-op query against the database. That makes it easy to drive from anywhere (command line, web app, GUI app, etc.)
If these are publicly available servers you can use a service like binarycanary.com to poll a page or service in your app.
I have got 1000 dedicated Java threads where each thread polls a corresponding url every one second.
public class Poller {
public static Node poll(Node node) {
GetMethod method = null;
try {
HttpClient client = new HttpClient(new SimpleHttpConnectionManager(true));
......
} catch (IOException ex) {
ex.printStackTrace();
} finally {
method.releaseConnection();
}
}
}
The threads are run every one second:
for (int i=0; i <1000; i++) {
MyThread thread = threads.get(i) // threads is a static field
if(thread.isAlive()) {
// If the previous thread is still running, let it run.
} else {
thread.start();
}
}
The problem is if I run the job every one second I get random exceptions like these:
java.net.BindException: Address already in use
INFO httpclient.HttpMethodDirector: I/O exception (java.net.BindException) caught when processing request: Address already in use
INFO httpclient.HttpMethodDirector: Retrying request
But if I run the job every 2 seconds or more, everything runs fine.
I even tried shutting down the instance of SimpleHttpConnectionManager() using shutDown() with no effect.
If I do netstat, I see thousands of TCP connections in TIME_WAIT state, which means they are have been closed and are clearing up.
So to limit the no of connections, I tried using a single instance of HttpClient and use it like this:
public class MyHttpClientFactory {
private static MyHttpClientFactory instance = new HttpClientFactory();
private MultiThreadedHttpConnectionManager connectionManager;
private HttpClient client;
private HttpClientFactory() {
init();
}
public static HttpClientFactory getInstance() {
return instance;
}
public void init() {
connectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams managerParams = new HttpConnectionManagerParams();
managerParams.setMaxTotalConnections(1000);
connectionManager.setParams(managerParams);
client = new HttpClient(connectionManager);
}
public HttpClient getHttpClient() {
if (client != null) {
return client;
} else {
init();
return client;
}
}
}
However after running for exactly 2 hours, it starts throwing 'too many open files' and eventually cannot do anything at all.
ERROR java.net.SocketException: Too many open files
INFO httpclient.HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Too many open files
INFO httpclient.HttpMethodDirector: Retrying request
I should be able to increase the no of connections allowed and make it work, but I would just be prolonging the evil. Any idea what is the best practise to use HttpClient in a situation like above?
Btw, I am still on HttpClient3.1.
This happened to us a few months back. First, double check to make sure you really are calling releaseConnection() every time. But even then, the OS doesn't actually reclaim the TCP connections all at once. The solution is to use the Apache HTTP Client's MultiThreadedHttpConnectionManager. This pools and reuses the connections.
See http://hc.apache.org/httpclient-3.x/performance.html for more performance tips.
Update: Whoops, I didn't read the lower code sample. If you're doing releaseConnection() and using MultiThreadedHttpConnectionManager, consider whether your OS limit on open files per process is set high enough. We had that problem too, and needed to extend the limit a bit.
There is nothing wrong with first error. You just depleted empirical ports available. Each TCP connection can stay in TIME_WAIT state for 2 minutes. You generate 2000/seconds. Soon or later, the socket can't find any unused local port and you will get that error. TIME_WAIT designed exactly for this purpose. Without it, your system might hijack a previous connection.
The second error means you have too many sockets open. On some system, there is a limit of 1K open files. Maybe you just hit that limit due to lingering sockets and other open files. On Linux, you can change this limit using
ulimit -n 2048
But that's limited by a system-wide max value.
As sudo or root edit the /etc/security/limits.conf file. At the end of the file just above “# End of File” enter the following values:
* soft nofile 65535
* hard nofile 65535
This will set the number of open files to unlimited.