We Keep Coding

UrlConnection API Call takes much more time the first time, then onwards it is comparable to curl.exe or postman

I have observed that one of my api is taking much more time if called through Java (URLConnection or Apache Http Client or OKHttp) for the first time. For the subsequent calls, the time is much lesser.
Although Postman or curl.exe takes very less time(comparable to the second iterations of java)
For my machine, the first time overhead is around 2 secs. But on some machines this is rising to around 5-6 secs for the first time. Thereafter it is around 300 ms roundtrip.
Here is my sample code:
public static String DoPostUsingURLConnection(String s_uri) throws Exception {
try {
URL uri = new URL(s_uri);
HttpURLConnection connection = (HttpURLConnection) uri.openConnection();
// Logger.log("Opened Connection");
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setDoOutput(true);
connection.setRequestProperty("Authorization", authorizationHeader);
// Create the Request Body
try (OutputStream os = connection.getOutputStream()) {
byte[] input = jsonRequestBody.getBytes("utf-8");
os.write(input, 0, input.length);
}
// Logger.log("Written Output Stream");
int responseCode = connection.getResponseCode();
InputStream is = null;
if (responseCode == HttpURLConnection.HTTP_OK)
is = connection.getInputStream();
else
is = connection.getErrorStream();
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine).append("\n");
;
}
in.close();
return response.toString();
} catch (Exception ex) {
return ex.getMessage();
} finally {
// Logger.log("Got full response");
}

You can investigate where time is taken by logging OkHttp connections events.
https://square.github.io/okhttp/events/
It will be particularly relevant if you are getting an IPv4 address and IPv6 and one is timing out and the other one succeeding.

This is just a guess. But the way Http connection works, that when you invoke it for the first time the connection gets established and that takes time. After that Http protocol doesn't really close connection for some time in expectation that some more requests would come and the connection could be re-used. And in your case indeed you send subsequent requests that re-use the previously created connection rather then re-establishing it which is expansive. I have written my own Open Source library that has a simplistic Http client in it. I noticed the same effect that first request takes much longer time than subsequent requests. But that doesn't explain why in Postman and curl we don't see the same effect. Anyway, if you want to solve this problem and you know your URL in advance, send a request upon your app initialization (you can even do it in separate thread). That will solve your problem.
If you are interested to look at my library here is Javadoc link. You can find it as maven artifact here and on github here. Article about the library covering partial list of features here

How to fix dialogflow authentication issue

I have a dialogflow project that I'm trying to access from Java with a rest call.
It is giving me an authentication issue.
I have followed all online instructions (and many forum suggestions) to no avail.
I have tried generating the key json, as per the instructions here:
https://dialogflow.com/docs/reference/v2-auth-setup
and setting my environment variable as described, but nothing seems to work.
I have checked my projectID, and am running off the same machine with the environment variable, and have double, triple and quadruple checked it's name and location, but I still get the following error:
java.net.HttpRetryException: cannot retry due to server authentication, in streaming mode
Here is my code (though it's a REST call, so I don't know if it's so relevant):
String url = https://dialogflow.googleapis.com/v2/projects/MYPROJECT/agent/sessions/SESSION_NUM:detectIntent
URL url = new URL(full_url);
String inText = "Hello World";
String outText = "";
HttpURLConnection con = (HttpURLConnection) url.openConnection();
try {
con.setDoOutput(true);
con.setRequestMethod("POST");
// set body of http post
Map<String,String> arguments = new HashMap<>();
JSONObject inTextJsn = new JSONObject();
inTextJsn.append("text",inText);
inTextJsn.append("languageCode","en");
JSONObject fieldJsn = new JSONObject();
fieldJsn.append("text", inTextJsn);
arguments.put("queryInput", fieldJsn.toString());
StringJoiner sj = new StringJoiner("&");
for(Map.Entry<String,String> entry : arguments.entrySet())
sj.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "="
+ URLEncoder.encode(entry.getValue(), "UTF-8"));
// post http post as bytes
byte[] bytes_out = sj.toString().getBytes(StandardCharsets.UTF_8);
con.setFixedLengthStreamingMode(bytes_out.length);
con.connect();
try (OutputStream os = con.getOutputStream()) {
os.write(bytes_out);
}
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream(),
"UTF-8"));
// read all lines to a string
String line;
String response = "";
while ((line = reader.readLine()) != null) {
response += line;
}
JSONObject responseJsn = new JSONObject(response);
outText = responseJsn.get("fulfillmentText").toString();
} catch (Exception e) {
System.err.println(e);
} finally {
con.disconnect();
}
return restResponse;
The gist of the code is to simply send a message ("Hello World!") to my dialogflow, and get back my agent's response (the code may have bugs - it's a bit hard to test when I can't get passed this authentication issue, so please help with the authentication, not code bugs).
Thanks all!

The directions at that page assume you're going to use the gcloud program to generate a currently valid bearer token, which is then sent along with the HTTP headers. That page illustrates
Your code doesn't seem to be generating an Authorization HTTP header at all, which is why you're getting the error you do.
Since you're using Java, you should look at the google-auth-library-java library, which will give you the tools to generate the token you need to provide in the Authorization header.
You may also wish to check out the google-cloud-java library. This contains Java classes to directly perform operations against Dialogflow instead of coding the REST/HTTP calls yourself. (However, it is still at an Alpha level for Dialogflow, so may not be stable or forwards compatible.)

How to Pass a File through an HttpURLConnection

I'm trying to get an image hosting on our server available to be displayed on a client. As per the specs of the project:
"When a Client receives such a URL, it must download the
contents (i.e., bytes) of the file referenced by the URL.
Before the Client can display the image to the user, it must first retrieve (i.e., download) the bytes of the
image file from the Server. Similarly, if the Client receives the URL of a known data file or a field help file
from the Server, it must download the content of those files before it can use them."
I'm pretty sure we have the server side stuff down, because if I put the url into a browser it retrieves and displays just fine. So it must be something with the ClientCommunicator class; can you take a look at my code and tell me what the problem is? I've spent hours on this.
Here is the code:
Where I actually call the function to get and display the file: (This part is working properly insofar as it is passing the right information to the server)
JFrame f = new JFrame();
JButton b = (JButton)e.getSource();
ImageIcon image = new ImageIcon(ClientCommunicator.DownloadFile(HOST, PORT, b.getLabel()));
JLabel l = new JLabel(image);
f.add(l);
f.pack();
f.setVisible(true);
From the ClientCommunicator class:
public static byte[] DownloadFile(String hostname, String port, String url){
String image = HttpClientHelper.doGetRequest("http://"+hostname+":"+port+"/"+url, null);
return image.getBytes();
}
The pertinent httpHelper:
public static String doGetRequest(String urlString,Map<String,String> headers){
URL url;
HttpURLConnection connection = null;
try {
//Create connection
url = new URL(urlString);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches (false);
connection.setDoInput(true);
connection.setDoOutput(true);
if(connection.getResponseCode() == 500){
return "failed";
}
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = rd.readLine()) != null) {
response.append(line);
}
rd.close();
return response.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if(connection != null) {
connection.disconnect();
}
}
}
After that, it jumps into the server stuff, which as I stated I believe is working correctly because clients such as Chrome can retrieve the file and display it properly. The problem has to be somewhere in here.
I believe that it has to do with the way the bytes are converted into a string and then back, but I do not know how to solve this problem. I've looked at similar problems on StackOverflow and have been unable to apply them to my situation. Any pointers in the right direction would be greatly appreciated.

If your server is sending binary data, you do not want to use an InputStreamReader, or in fact a Reader of any sort. As the Java API indicates, Readers are for reading streams of characters (not bytes) http://docs.oracle.com/javase/7/docs/api/java/io/Reader.html, which means you will run into all sorts of encoding issues.
See this other stack overflow answer for how to read bytes from a stream:
Convert InputStream to byte array in Java

Do your homework.
Isolate the issue. Modify the server side to send only 256 all possible bytes. Do a binary search and reduce it to small set of bytes.
Use http proxy tools to monitor the bytes as they are transmitted. Fiddler in windows world. Find other ones for the *nix environments.
Then see where the problem is happening and google/bing the suspicions or share the result.

Servlet that sends back JSON: Confusion on reception

I have a Servlet that sends back a JSON Object and I would like to use this servlet in another Java project. I have this method that gets me the results:
public JSONArray getSQL(String aServletURL)
{
JSONArray toReturn = null;
String returnString = "";
try
{
URL myUrl = new URL(aServletURL);
URLConnection conn = myUrl.openConnection();
conn.setDoOutput(true);
BufferedReader in = new BufferedReader( new InputStreamReader( conn.getInputStream() ) );
String s;
while ((s = in.readLine()) != null )
returnString += s;
in.close();
toReturn = new JSONArray(returnString);
}
catch(Exception e)
{
return new JSONArray();
}
return toReturn;
}
This works pretty will, but the problem I am facing is the following:
When I do several simultaneous requests, the results get mixed up and I sometimes get a Response that does not match the request I send.
I suspect the problem to be related to the way I get the response back: The Reader reading a String from the InputStream of the connection.
How can I make sure that I get one reques -> one corresponding reply ?
Is there a better way to retrieve my JSON object from my servlet ?
Cheers,
Tim

When I do several simultaneous requests, the results get mixed up and I sometimes get a Response that does not match the request I send.
Your servlet is not thread safe. I'd bet that you've improperly assigned request scoped data either directly or indirectly as instance or class variables of the servlet. This is a common beginner's mistake.
Carefully read this How do servlets work? Instantiation, sessions, shared variables and multithreading and fix your servlet code accordingly. The problem is not in the URLConnection code shown so far, although it indicates that you're doing exactly the same job in both doGet() and doPost(), which in turn is already a smell as to how the servlet is designed.

Try removing setDoOutput(true), you are using the connection only for input and so you shouldn't use it.
Edit: alternatively try using HttpClient, it's much nicer that using "raw" Java.

HttpsURLConnection Connection Problems

I'm a problem with a HttpsURLConnection that I can't seem to solve. Basically, I'm sending up some info to a server and if some of that data is wrong, the server sends me a 500 response code. However, it also sends a message in the response telling me which bit of data was wrong. The problem is that the message is always empty when I read it in. I think this is because a filenotfound exception always gets thrown before the stream can be read. Am I right? I tried reading the errorstream as well but this is always empty. Here's a snippet:
conn = (HttpsURLConnection) connectURL.openConnection();
conn.setDoOutput(true);
conn.setConnectTimeout(30000);
conn.setReadTimeout(30000);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Length",
Integer.toString(outString.getBytes().length));
DataOutputStream wr = new DataOutputStream(conn
.getOutputStream());
wr.write(outString.getBytes());
wr.flush();
wr.close();
if(conn.getResponseCode>400{
String response = getErrorResponse(conn);
public String getErrorResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
//is = conn.getInputStream();
is = conn.getErrorStream();
// scoop up the reply from the server
int ch;
StringBuffer sb = new StringBuffer();
while ((ch = is.read()) != -1) {
sb.append((char) ch);
}
//System.out.println(sb.toString());
return sb.toString();
// return conferenceId;
}
catch (Exception e){
e.printStackTrace();
}
}

So just to follow up on this, here is how I solved it:
public static String getResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
if(conn.getResponseCode()>=400){
is = conn.getErrorStream();
}
else{
is=conn.getInputStream();
}
...read stream...
}
It seems that calling them like this produced an error stream with a message. Thanks for the suggestions!

Try setting content-type request property to "application/x-www-form-urlencoded"
The same is mentioned on this link:
http://developers.sun.com/mobility/midp/ttips/HTTPPost/
The Content-Length and Content-Type headers are critical because they tell the web server how many bytes of data to expect, and what kind, identified by a MIME type.
In MIDP clients the two most popular MIME types are application/octet-stream, to send raw binary data, and application/x-www-form-urlencoded, to send name-value pairs

Are you in control of the server? In other words, did you write the process that runs on the server and listens to the port you're trying to access?
If you did, then you should also be able to debug it and see why your process returns 404.
If you didn't, then describe your architecture (HTTP server, the component it invokes to respond to your HTTP(S) request, etc) and we'll take it from there.
In the very simplest case, of an HTTP server being an Apache server yielding control to some PHP script, it means that Apache couldn't assign your request to anything. Most likely a Web server misconfiguration. Provide some more details and we'll help you out.