We Keep Coding

UrlConnection API Call takes much more time the first time, then onwards it is comparable to curl.exe or postman

I have observed that one of my api is taking much more time if called through Java (URLConnection or Apache Http Client or OKHttp) for the first time. For the subsequent calls, the time is much lesser.
Although Postman or curl.exe takes very less time(comparable to the second iterations of java)
For my machine, the first time overhead is around 2 secs. But on some machines this is rising to around 5-6 secs for the first time. Thereafter it is around 300 ms roundtrip.
Here is my sample code:
public static String DoPostUsingURLConnection(String s_uri) throws Exception {
try {
URL uri = new URL(s_uri);
HttpURLConnection connection = (HttpURLConnection) uri.openConnection();
// Logger.log("Opened Connection");
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setDoOutput(true);
connection.setRequestProperty("Authorization", authorizationHeader);
// Create the Request Body
try (OutputStream os = connection.getOutputStream()) {
byte[] input = jsonRequestBody.getBytes("utf-8");
os.write(input, 0, input.length);
}
// Logger.log("Written Output Stream");
int responseCode = connection.getResponseCode();
InputStream is = null;
if (responseCode == HttpURLConnection.HTTP_OK)
is = connection.getInputStream();
else
is = connection.getErrorStream();
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine).append("\n");
;
}
in.close();
return response.toString();
} catch (Exception ex) {
return ex.getMessage();
} finally {
// Logger.log("Got full response");
}

You can investigate where time is taken by logging OkHttp connections events.
https://square.github.io/okhttp/events/
It will be particularly relevant if you are getting an IPv4 address and IPv6 and one is timing out and the other one succeeding.

This is just a guess. But the way Http connection works, that when you invoke it for the first time the connection gets established and that takes time. After that Http protocol doesn't really close connection for some time in expectation that some more requests would come and the connection could be re-used. And in your case indeed you send subsequent requests that re-use the previously created connection rather then re-establishing it which is expansive. I have written my own Open Source library that has a simplistic Http client in it. I noticed the same effect that first request takes much longer time than subsequent requests. But that doesn't explain why in Postman and curl we don't see the same effect. Anyway, if you want to solve this problem and you know your URL in advance, send a request upon your app initialization (you can even do it in separate thread). That will solve your problem.
If you are interested to look at my library here is Javadoc link. You can find it as maven artifact here and on github here. Article about the library covering partial list of features here

Java HTTPUrlConnection returns 500 status code

I'm trying to GET a url using HTTPUrlConnection, however I'm always getting a 500 code, but when I try to access that same url from the browser or using curl, it works fine!
This is the code
try{
URL url = new URL("theurl");
HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
httpcon.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpcon.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:14.0) Gecko/20100101 Firefox/14.0.1");
System.out.println(httpcon.getHeaderFields());
}catch (Exception e) {
System.out.println("exception "+e);
}
When I print the headerfields, it shows the 500 code.. when I change the URL to something else like google.com , it works fine. But I don't understand why it doesn't work here but it works fine on the browser and with curl.
Any help would be highly appreciated..
Thank you,

This is mostly happening because of encoding.
If you are using browser OK, but getting 500 ( internal server error ) in your program,it is because the browsers have a highly sophisticated code regarding charsets and content-types.
Here is my code and it works in the case of ISO8859_1 as charset and english language.
public void sendPost(String Url, String params) throws Exception {
String url=Url;
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
con.setRequestProperty("Acceptcharset", "en-us");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("charset", "EN-US");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
String urlParameters=params;
// Send post request
con.setDoOutput(true);
con.setDoInput(true);
con.connect();
//con.
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
this.response=response.toString();
con.disconnect();
}
and in the main program , call it like this:
myclassname.sendPost("https://change.this2webaddress.desphilboy.com/websitealias/orwebpath/someaction","paramname="+URLEncoder.encode(urlparam,"ISO8859_1"))

The status code 500 suggests that the code at web server have been crashed .Use HttpURLConnection#getErrorStream() to get more idea of the error. Refer Http Status Code 500

I ran into the problem of "URL works in browser, but when I do http-get in java I get a 500 Error".
In my case the problem was that the regular http-get ended up in an infinite redirect loop between /default.aspx and /login.aspx
URL oUrl = new URL(url);
HttpURLConnection con = (HttpURLConnection) oUrl.openConnection();
con.setRequestMethod("GET");
...
int responseCode = con.getResponseCode();
What was happening was: The server serves up a three-part cookie and con.getResponseCode() only used one of the parts. The cookie data in the header looked like this:
header.key = null
value = HTTP/1.1 302 Found
...
header.key = Location
value = /default.aspx
header.key = Set-Cookie
value = WebCom-lbal=qxmgueUmKZvx8zjxPftC/bHT/g/rUrJXyOoX3YKnYJxEHwILnR13ojZmkkocFI7ZzU0aX9pVtJ93yNg=; path=/
value = USE_RESPONSIVE_GUI=1; expires=Wed, 17-Apr-2115 18:22:11 GMT; path=/
value = ASP.NET_SessionId=bf0bxkfawdwfr10ipmvviq3d; path=/; HttpOnly
...
So the server when receiving only a third of the needed data got confused: You're logged in! No wait, you have to login. No, you're logged in, ...
To work around the infinite redirect-loop I had to manually look for re-directs and manually parse through the header for "Set-cookie" entries.
con = (HttpURLConnection) oUrl.openConnection();
con.setRequestMethod("GET");
...
log.debug("Disable auto-redirect. We have to look at each redirect manually");
con.setInstanceFollowRedirects(false);
....
int responseCode = con.getResponseCode();
With this code the parsing of the cookie, if we get a redirect in the responseCode:
private String getNewCookiesIfAny(String origCookies, HttpURLConnection con) {
String result = null;
String key;
Set<Map.Entry<String, List<String>>> allHeaders = con.getHeaderFields().entrySet();
for (Map.Entry<String, List<String>> header : allHeaders) {
key = header.getKey();
if (key != null && key.equalsIgnoreCase(HttpHeaders.SET_COOKIE)) {
// get the cookie if need, for login
List<String> values = header.getValue();
for (String value : values) {
if (result == null || result.isEmpty()) {
result = value;
} else {
result = result + "; " + value;
}
}
}
}
if (result == null) {
log.debug("Reuse the original cookie");
result = origCookies;
}
return result;
}

Make sure that your connection allows following redirects - this is one of the possible reasons for difference in behaviour between your connection and the browser (allows redirect by default).
It should be returning code 3xx, but there maybe something else somewhere that changes it to 500 for your connection.

I faced the same issue, and our issue was there was a special symbol in one of the parameter values. We fixed it by using URLEncoder.encode(String, String)

In my case it turned out that the server always returns HTTP/1.1 500 (in Browser as in Java) for the page I wanted to access, but successfully delivers the webpage content nonetheless.
A human accessing the specific page via Browser just doesn't notice, since he will see the page and no error message, in Java I had to read the error stream instead of the input stream (thanks #Muse).
I have no idea why, though. Might be some obscure way to keep Crawlers out.

This is an old question, but I have had same issue and solved it this way.
This might help other is same situation.
In my case I was developing system on local environment, and every thing worked fine when I checked my Rest Api from browser but I got all the time thrown HTTP error 500 in my Android system.
The problem is when you work on Android, it works on VM (Virtual Machine), that said it means your local computer firewall might preventing your Virtual Machine accessing the local URL (IP) address.
You need just to allow that in your computer firewall. The same thing apply if you trying to access system from out side your network.

Check the parameter
httpURLConnection.setDoOutput(false);
Only for GET Method and set to true on POST, this save me lot of time!!!

Android HttpUrlConnection Response wait time

I have below code. I am getting HTTP response code -1. Just to troubleshoot, I would like to know if I should wait after making connection for a while before checking the http response code.
String requestURL = "https://www.google.com";
HttpURLConnection connection = new HttpURLConnection;
connection = (HttpURLConnection) requestUrl.openConnection();
connection.setConnectTimeout(20000);
connection.setReadTimeout(20000);
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setDoInput(true);
connection.connect();
OutputStreamWriter writer = new OutputStreamWriter(this.connection.getOutputStream());
writer.write(getHttpData());
writer.flush();
writer.close();
if(connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
BufferedReader reader = new BufferedReader(new inputStreamReader(connection.getInputStream()));
}
Is getResponseCode() or getInputStream() blocking call ? Am I reading response too soon ? Should I wait ?
Thanks for the help.

One reason you could be getting the weird HTTP response code of -1 is the connection-pooling with-http-keep-alives bug that's in Android platform versions up to (and including) Froyo, ie Android v2.2.
The Android Developer Blog gives the following code snippet to resolve the problem (note that they say it doesn't include Froyo, although we found that it did - the snippet below is modified accordingly):
private void disableConnectionReuseIfNecessary() {
// HTTP connection reuse which was buggy pre-froyo
if (Integer.parseInt(Build.VERSION.SDK) <= Build.VERSION_CODES.FROYO) {
System.setProperty("http.keepAlive", "false");
}
}
Try calling that when your app starts-up and see if it fixes your problem.

HttpsURLConnection Connection Problems

I'm a problem with a HttpsURLConnection that I can't seem to solve. Basically, I'm sending up some info to a server and if some of that data is wrong, the server sends me a 500 response code. However, it also sends a message in the response telling me which bit of data was wrong. The problem is that the message is always empty when I read it in. I think this is because a filenotfound exception always gets thrown before the stream can be read. Am I right? I tried reading the errorstream as well but this is always empty. Here's a snippet:
conn = (HttpsURLConnection) connectURL.openConnection();
conn.setDoOutput(true);
conn.setConnectTimeout(30000);
conn.setReadTimeout(30000);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Length",
Integer.toString(outString.getBytes().length));
DataOutputStream wr = new DataOutputStream(conn
.getOutputStream());
wr.write(outString.getBytes());
wr.flush();
wr.close();
if(conn.getResponseCode>400{
String response = getErrorResponse(conn);
public String getErrorResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
//is = conn.getInputStream();
is = conn.getErrorStream();
// scoop up the reply from the server
int ch;
StringBuffer sb = new StringBuffer();
while ((ch = is.read()) != -1) {
sb.append((char) ch);
}
//System.out.println(sb.toString());
return sb.toString();
// return conferenceId;
}
catch (Exception e){
e.printStackTrace();
}
}

So just to follow up on this, here is how I solved it:
public static String getResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
if(conn.getResponseCode()>=400){
is = conn.getErrorStream();
}
else{
is=conn.getInputStream();
}
...read stream...
}
It seems that calling them like this produced an error stream with a message. Thanks for the suggestions!

Try setting content-type request property to "application/x-www-form-urlencoded"
The same is mentioned on this link:
http://developers.sun.com/mobility/midp/ttips/HTTPPost/
The Content-Length and Content-Type headers are critical because they tell the web server how many bytes of data to expect, and what kind, identified by a MIME type.
In MIDP clients the two most popular MIME types are application/octet-stream, to send raw binary data, and application/x-www-form-urlencoded, to send name-value pairs

Are you in control of the server? In other words, did you write the process that runs on the server and listens to the port you're trying to access?
If you did, then you should also be able to debug it and see why your process returns 404.
If you didn't, then describe your architecture (HTTP server, the component it invokes to respond to your HTTP(S) request, etc) and we'll take it from there.
In the very simplest case, of an HTTP server being an Apache server yielding control to some PHP script, it means that Apache couldn't assign your request to anything. Most likely a Web server misconfiguration. Provide some more details and we'll help you out.

URLConnection sometimes returns empty string response?

I'm making an http GET request. It works in about 70% of my attempts. For some reason, I sometimes get no response string from a successful connection. I just setup a button in my app which keeps firing the code below. One call might fail to reply with a string, the next call works fine:
private onButtonClick() {
try {
doit();
} catch (Exception ex) {
...
}
}
public void doit() throws Exception {
URL url = new URL("http://www.example.com/service");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setDoInput(true);
connection.setUseCaches(false);
connection.setAllowUserInteraction(false);
connection.setReadTimeout(30 * 1000);
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setRequestProperty("Authorization",
"Basic " + Base64.encode("username" + ":" + "password"));
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = "";
StringBuilder sb = new StringBuilder();
while ((line = in.readLine()) != null) {
sb.append(line);
}
in.close();
connection.disconnect();
// Every so often this prints an empty string!
System.out.println(sb.toString());
}
am I doing something wrong here? It seems like maybe I'm not closing the connection properly from the last call somehow and the response gets mangled or something? I am also calling doit() from multiple threads simultaneously, but I thought the contents of the method are thread-safe, same behavior though,
Thanks
Thanks

That method looks fine. It's reentrant, so calls shouldn't interfere with each other. It's probably a server issue, either deliberate throttling or just a bug.
EDIT: You can check the status code with getResponseCode.

For checking ResponseCode:
BufferedReader responseStream;
if (((HttpURLConnection) connection).getResponseCode() == 200) {
responseStream = new BufferedReader(new InputStreamReader(connection.getInputStream(), "UTF-8"));
} else {
responseStream = new BufferedReader(new InputStreamReader(((HttpURLConnection) connection).getErrorStream(), "UTF-8"));
}
For empty content resposneCode is 204. So if u can get empty body just add one more "if" with 204 code.

We also came across with the similar scenario, I came across the following solution for this issue:
- Setting up a user agent string on URLConnection object.
URLConnection conn = url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)");
more details