I want to get the file size of a file on a remote connection without actually downloading the (large) file. I am using the "Content-Length" header of the file. The relevant code is:
URL obj = new URL(FILES_URL + fileName);
String contentLength = "";
HttpURLConnection conn = null;
try {
conn = (HttpURLConnection) obj.openConnection();
conn.setConnectTimeout(3000);
conn.setReadTimeout(3000);
contentLength = conn.getHeaderField("Content-Length");
int responseCode = conn.getResponseCode();
Log.d(TAG, "responseCode: " + responseCode);
} finally {
Log.d(TAG, "pre-disconnect");
if (conn!=null) conn.disconnect();
Log.d(TAG, "post-disconnect");
}
return contentLength;
The command "conn.disconnect();" sometimes seems to take forever. I have seen 23 seconds! Admittedly, this is connecting to a secondary local device which is running a web server, but the WiFi signal is strong, relatively fast, and I have never had any such problems using "curl" from my laptop. I do not have control over the web server I am connecting too.
The problem possibly is enhanced when making multiple similar connections to different files one after another, not sure. This is, however, creating entirely new HttpURLConnection's and not reusing the old one. Could reusing the connection help?
I never actually download the file or access the inputstream.
I could just not call disconnect, but I understand it is not recommended because resources would not be released. Is this not correct? I notice URLConnection doesn't have a disconnect. It is just suggested to close any streams you open.
This code is in an asynctask. I guess I could try moving the disconnect call itself to a further asynctask because I don't do anything afterwards. Not sure if that is even possible.
Do you have any suggestions? Should I try something other than HttpURLConnection to get the file size without downloading the file?
Thanks to EJP in the comments. Changing the request method to "HEAD" made the disconnect almost instantaneous:
conn.setRequestMethod("HEAD");
From what I have read, HttpURLConnection.disconnect() will skip through the entire response object if it hasn't been read. Therefore, for very large files, it will take a long time. Using the request method "HEAD" force the response body to be empty and solves the issue.
I suggest you to use either Volley or Okhttp for faster networking but depending on your requirement . Got through Comparison Of Volley And OkHttp and Retrofit and decide which library to use.
As suggestion if you putting this code inside AsyncTask then Read Dark Side of AsyncTask.
Related
So I have a problem with a Java program I have. The program's basic functionality includes basically connecting to a web API for data. The function that does that is something like this:
public static Object getData(String sURL) throws IOException {
URL url = new URL(sURL);
URLConnection request = url.openConnection();
request.connect();
return request.getContent();
}
The code works fine as it is, but recently, after my house changed ISPs, I have found that sometimes the connections take an unreasonably long amount of time, something like 10 seconds or more in about 10% of attempts, while the other 90% takes only around 200ms. I have found it to be faster to ask my program to call the function again in a different thread than to wait for some of these connections to finally connect.
Therefore, I want to change the function so that if after 500ms, the connection did not establish, it would disconnect and a new connection would be attempted. How could I do this?
Somewhere online I read that HttpURLConnection might help, but I am not sure how.
URLConnection allows you to specify the connect and read timeout prior to calling connect():
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/URLConnection.html#setConnectTimeout(int)
Sets a specified timeout value, in milliseconds, to be used when
opening a communications link to the resource referenced by this
URLConnection. If the timeout expires before the connection can be
established, a java.net.SocketTimeoutException is raised. A timeout of
zero is interpreted as an infinite timeout.
With 500ms timeout:
try {
URLConnection request = url.openConnection();
request.setConnectTimeout(500); // 500 ms
request.connect();
// on successful connection
} catch (SocketTimeoutException ex) {
// on request timeout
}
This you can pack into a loop, but I recommend limiting the number of attempts made.
Java's URLConnection doesn't have retry capabilities in Java 8 therefore the best way here to achieve this - use an appropriate standalone 3-party library such as Apache HttpClient.
This is by far the best standalone 3-party HTTP client with advanced capabilities as of 2020 and it's still maintained.
By default as of version 5.2.x Apache Http Client, Apache Http Client uses the default implementation of org.apache.http.client.HttpRequestRetryHandler, which retries 3 times, but you can use a custom implementation instead.
The configuration might look like this(full imports are for example's sake):
org.apache.http.client.HttpClient httpClient = org.apache.http.impl.client.HttpClients.custom()
.setRetryHandler(YourCustomImplOfTheRetryHandlerClass)
//other config
.build();
There is no way I can reproduce that problem using my ISP.
I suggest you dig deeper into the problem and find a better solution. Sending another request just doesn't seem good enough to me. Maybe try a different way to get the data and see if that works for you. Can't say for sure as I can't reproduce the problem.
Edit:
As I've just seen, it happens even with the simplest setup:
InputStream stream = new URL("http://xx.xx.xxx.xxx/GetAll.php").openStream();
Gives the same timeout error. I think I'm missing some basic configuration.
I used HTTPGet to connect to a PHP web service I have.
I saw it's deprecated so I've been trying to switch to the recommended HttpUrlConnection but with no success.
The HttpURLConnection does not seem to be able connect to the service, even though I can connect from my web browser without any problem.
My connection code:
URL myUrl = new URL("http://xx.xx.xxx.xxx/GetAll.php");
HttpURLConnection request = (HttpURLConnection)myUrl.openConnection();
request.setRequestProperty("Content-Type","text/xml;charset=UTF-8");
InputStream stream = request.getInputStream();
The GetAll.php file:
<?
require_once('MysqliDb.php'); //Helper class
$db = new MysqliDb();
//All closest events by date
$All = $db->query("SELECT * FROM Event;");
//Return in JSON
echo json_encode($All);
The result I am getting from the file:
[{"EventID":1,"StartTime":1300,"Duration":1,"EventDate":"2015-05-17","EventOrder":1,"Type":0,"Name":"\u05e2\u05d1\u05e8\u05d9\u05ea AND ENGLISH","Organiser":"Neta","Phone":"012345678","Location":"Loc","Description":"Desc"}]
Thank you,
Neta
I want to share my solution, as this has cost me hours of hair tearing.
As it turns out, "Timed out" exception has nothing to do with the code, it's a network connectivity issue. The phone I used to debug the app sometimes appears to be connected to Wifi even though it really isn't.
Anyway, if you have this exception, try checking your network connection.
Good luck!
I am writing an application in which I am using Http Connection to post data on a server. My App is working fine when I check it in emulator. My Webservice take too much time to generating responses and my emulator is also responding in a proper way. Somehow when I installed the app on the device, my app is posting data twice on the server. I have checked it... Does anyone have any solution on how to escape from this???
Here I am attaching my code of the sending request. I think the mobile app is sending another request when it reaches HTTP Time out, but I don't know what's the problem. Please Help me.
String param=
"function=OpenRecharge&LoginId="+SharedVariable.getUserInfo().getLoginID()
+"&BatchId="+SharedVariable.getSelectedProduct().getBatchID()
+"&SystemServiceID="+SharedVariable.getSelectedProduct().getSystemServiceID()
+"&ReferalNumber="+strMobileNo
+"&Amount="+strAmount
+"&FromANI="+fromMoNo
+"&Email="
+"&Checksum="+Checksum;
System.out.println("Final Parameter:\n"+param);
connection = (HttpConnection) Connector.open(url);
//Connector.open(param, strAmount, quit)
connection.setRequestMethod(HttpConnection.POST);
connection.setRequestProperty("User-Agent", "Profile/MIDP-1.0 Configuration/CLDC-1.0");
connection.setRequestProperty("Accept_Language","en-US");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
OutputStream out = connection.openOutputStream();
out.write(param.getBytes());
out.flush();
//System.out.println("Status Line Code: " + connection.getResponseCode());
//System.out.println("Status Line Message: " + connection.getResponseMessage());
is=connection.openDataInputStream();
int chr;
StringBuffer sb=new StringBuffer();
while ((chr = is.read()) != -1)
sb.append((char) chr);
System.out.println("Response===>"+sb.toString());
Can you serve up an answer with a place holder text like, "processing..." then after a delay have the web browser try again?
I´m not app developer, anyway your web sevice shouldn´t have a big delay for response. I think this is your problem and you should resolve it. Making a cache or pre-process your response.
Although you can change timeout (it seems fixed), it´s not recommendable beacuse many mobile (wap) proxies have timeout of 30s.
When using HttpURLConnection does the InputStream need to be closed if we do not 'get' and use it?
i.e. is this safe?
HttpURLConnection conn = (HttpURLConnection) uri.getURI().toURL().openConnection();
conn.connect();
// check for content type I don't care about
if (conn.getContentType.equals("image/gif") return;
// get stream and read from it
InputStream is = conn.getInputStream();
try {
// read from is
} finally {
is.close();
}
Secondly, is it safe to close an InputStream before all of it's content has been fully read?
Is there a risk of leaving the underlying socket in ESTABLISHED or even CLOSE_WAIT state?
According to http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
and OpenJDK source code.
(When keepAlive == true)
If client called HttpURLConnection.getInputSteam().close(), the later call to HttpURLConnection.disconnect() will NOT close the Socket. i.e. The Socket is reused (cached)
If client does not call close(), call disconnect() will close the InputStream and close the Socket.
So in order to reuse the Socket, just call InputStream.close(). Do not call HttpURLConnection.disconnect().
is it safe to close an InputStream
before all of it's content has been
read
You need to read all of the data in the input stream before you close it so that the underlying TCP connection gets cached. I have read that it should not be required in latest Java, but it was always mandated to read the whole response for connection re-use.
Check this post: keep-alive in java6
Here is some information regarding the keep-alive cache. All of this information pertains Java 6, but is probably also accurate for many prior and later versions.
From what I can tell, the code boils down to:
If the remote server sends a "Keep-Alive" header with a "timeout" value that can be parsed as a positive integer, that number of seconds is used for the timeout.
If the remote server sends a "Keep-Alive" header but it doesn't have a "timeout" value that can be parsed as a positive integer and "usingProxy" is true, then the timeout is 60 seconds.
In all other cases, the timeout is 5 seconds.
This logic is split between two places: around line 725 of sun.net.www.http.HttpClient (in the "parseHTTPHeader" method), and around line 120 of sun.net.www.http.KeepAliveCache (in the "put" method).
So, there are two ways to control the timeout period:
Control the remote server and configure it to send a Keep-Alive header with the proper timeout field
Modify the JDK source code and build your own.
One would think that it would be possible to change the apparently arbitrary five-second default without recompiling internal JDK classes, but it isn't. A bug was filed in 2005 requesting this ability, but Sun refused to provide it.
If you really want to make sure that the connection is close you should call conn.disconnect().
The open connections you observed are because of the HTTP 1.1 connection keep alive feature (also known as HTTP Persistent Connections).
If the server supports HTTP 1.1 and does not send a Connection: close in the response header Java does not immediately close the underlaying TCP connection when you close the input stream. Instead it keeps it open and tries to reuse it for the next HTTP request to the same server.
If you don't want this behaviour at all you can set the system property http.keepAlive to false:
System.setProperty("http.keepAlive","false");
When using HttpURLConnection does the InputStream need to be closed if we do not 'get' and use it?
Yes, it always needs to be closed.
i.e. is this safe?
Not 100%, you run the risk of getting a NPE. Safer is:
InputStream is = null;
try {
is = conn.getInputStream()
// read from is
} finally {
if (is != null) {
is.close();
}
}
You also have to close error stream if the HTTP request fails (anything but 200):
try {
...
}
catch (IOException e) {
connection.getErrorStream().close();
}
If you don't do it, all requests that don't return 200 (e.g. timeout) will leak one socket.
Since Java 7 the recommended way is
try (InputStream is = conn.getInputStream()) {
// read from is
// ...
}
as for all other classes implementing Closable. close() is called at the end of the try {...} block.
Closing the input stream also means you are done with reading. Otherwise the connection hangs around until the finalizer closes the stream.
Same applies to the output stream, if you are sending data.
There is no need to get an close the ErrorStream. Even if it implements the InputStream interface: It's using the InputStream in combination with a buffer. Closing the InputStream is sufficient.
I am looking to optimize a process that runs continually and makes frequent calls (> 1 per second on average) to an external API via a simple REST style HTTP post. One thing I've noticed is that currently, the HttpUrlConnection is created and closed for every API call, as per the following structure (non essential code and error handling removed for readability).
//every API call
try {
URL url = new URL("..remote_site..");
conn = (HttpURLConnection) url.openConnection();
setupConnectionOptions(conn); //sets things like timeoout and usecaches false
outputWriter = new OutputStreamWriter(new BufferedOutputStream(conn.getOutputStream()));
//send request
} finally {
conn.disconnect();
outputWriter.close();
}
I don't have extensive experience dealing with the http protocol directly, but based on common sense / knowledge of sockets in general it seems that it would be much more efficient to only create the connection once and re-use it, and only reinitialize it on a problem, to avoid the connection negotiation each time, like this:
//on startup, or error
private void initializeConnection()
{
URL url = new URL("..remote_site..");
conn = (HttpURLConnection) url.openConnection();
setupConnectionOptions(conn); //sets things like timeoout and usecaches false
}
//per request
try {
outputWriter = new OutputStreamWriter(new BufferedOutputStream(conn.getOutputStream()));
//send request
} catch (IOException) {
try conn.disconnect();
initializeConnection();
} finally {
outputWriter.close();
}
//on graceful exit
conn.disconnect();
My questions are:
is this a reasonable optimization in general (will the speed increase be noticeable)?
Assuming yes:
should I reuse the output stream as well the connection?
is it reasonable to only reinitialize connection on error, or should I do it after a certain number of requests / time?
Basically, yes, and it saves a lot of time --- setting up a socket takes significant effort, even worse with SSL. That's why "keepalive" was implemented back in the Old Days. That's a litle bit counter to the REST philosophy, but it's a performance optimization.
The one thing about it is that sockets are a limited resource; in a really heavy-use environment, you could end up with no sockets left for new connections. this is a Bad Thing.