We Keep Coding

Java executing curl command using HttpURLConnection returns 204 (HTTP_NO_CONTENT)

I am using Java 11. I have the the following curl command, when I execute on the command line:
curl --location --request GET 'http://xxx.xxx.co.za:8080/document/details/Select+docId+From+%27Workflow+Club%2FCustomer+Invoices%27+where+recursive+%3D+true+and+invoice_number+%3D%271221669023%27' --header 'Authorization: Basic xxx'
Returns the following:
{errorMessage: 'PaperTrail API only available in enterprise edition'}
However, when I try execute the same URL in a Java application using HttpURLConnection, it returns a blank response.
private static final String USER_AGENT = "Mozilla/5.0";
private static final String GET_URL = "http://xxx.xxx.co.za:8080/document/details/";
private static final String GET_URL_QRY = "Select docId From 'Workflow Club/Customer Invoices' where recursive = true and invoice_number =':1'";
private static final String GET_AUTH_ENC = "Basic xxx";
#Override
public String getDocId(Long invoiceNumber) {
String get_url_qry = StringUtils.replace(GET_URL_QRY, ":1", Long.toString(invoiceNumber));
get_url_qry = URLEncoder.encode(get_url_qry, StandardCharsets.UTF_8);
final String get_url = GET_URL+get_url_qry;
try {
URL url = new URL(get_url);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestProperty("Authorization", GET_AUTH_ENC);
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = con.getResponseCode();
logger.info(get_url+" -> GET Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK || responseCode == HttpURLConnection.HTTP_NO_CONTENT) { // success
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
String resp = response.toString();
logger.info(responseCode+" Response: '"+resp+"'.");
} else {
logger.error("GET request did not work (responseCode: "+responseCode+").");
}
} catch (MalformedURLException e) {
logger.error("MalformedURLException creating URL '"+get_url+"'. "+e.getMessage());
} catch (IOException e) {
logger.error("IOException creating connection from URL '"+get_url+"'. "+e.getMessage());
}
return null;
}
Outputs the following with a blank response:
204 Response: ''.
Question
How do I get the Java application to also return the same as the command line call?
UPDATE
I have a different POST url, that I need to call too, and I can call it successfully. So there's something wrong with my GET call.
private static final String USER_AGENT = "Mozilla/5.0";
E.g. GET call that returns a 204, with no content.
private String getDocId(Long invoiceNumber) {
String get_url_qry = StringUtils.replace(GET_URL_QRY, ":1", Long.toString(invoiceNumber));
get_url_qry = URLEncoder.encode(get_url_qry, StandardCharsets.UTF_8);
final String get_url = GET_URL+get_url_qry;
try {
URL url = new URL(get_url);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestProperty("Authorization", GET_AUTH_ENC);
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
Map<String,String> data = handleResponse(con);
return data.get("docId");
} catch (MalformedURLException e) {
logger.error("MalformedURLException creating URL '"+get_url+"'. "+e.getMessage());
} catch (IOException e) {
logger.error("IOException creating connection from URL '"+get_url+"'. "+e.getMessage());
}
return null;
}
The POST call, that returns a 200, and the expected content.
private String getDocLink(String docId) {
if (StringUtils.isNotBlank(docId)) {
try {
URL url = new URL(POST_URL);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestProperty("Authorization", GET_AUTH_ENC);
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
byte[] postDataBytes = getPostData(docId);
con.setRequestProperty("Content-Length", String.valueOf(postDataBytes.length));
con.setDoOutput(true);
con.getOutputStream().write(postDataBytes);
Map<String,String> data = handleResponse(con);
return data.get("url");
} catch (IOException e) {
logger.error("IOException creating connection from URL '"+POST_URL+"'. "+e.getMessage());
}
} else {
logger.error("No docId provided when trying to get a document link.");
}
return null;
}
So seeing the that POST call works, I think I must be doing something wrong with the GET call.

Did you try, setting the same user agent in your Java Code, cURL would use? something like curl/7.37.0?
As far as I can tell, that should be all, what differs. Aside cURL following redirects. But as there is no redirect, I guess it might be the User Agent making a difference.
There are a lot of server applications, behaving differently, when they are called by a browser (Like you make it think by setting the User-Agent to Mozilla/5.0), instead of some other application, like cURL.

From what you describe, the original call produces an error.
Assuming that you are also getting some form of error in the java code of your GET, you will catch the exception and simply log it. Then you will return null instead of a string, which will cause your response to have no content, i.e. 204.

How to get device Access token and perform multiple HTTP POST Request

I'm currently working on a project using an IoT platform " Thingsboard " where I've have created multiple devices, and I want to send data to each one of the devices from a JSON File, I'm using Rest Api to perform this request, but I've struggling for a while how to get the access token of my devices and parse each one of them in my request as a header param. I was just doing manually by getting them with Curl, but I want now to do it automatically. I know that Thingsboard has a Rest client Api written in java (https://thingsboard.io/docs/reference/rest-client/) so I've tried to use that in my script but I's not working. I'm new to working with Rest Api so if anybody can gie me a clue it would be so helpful.
here's a part of my code for the requests :
private static String token;
public String getToken() {
return token;
}
String paramValue = "param\\with\\backslash";
String yourURLStr = "http://host.com?param=" + java.net.URLEncoder.encode(paramValue, "UTF-8");
URL url2 = new URL("https://demo.thingsboard.io/api/v1/token/telemetry?token=$JWT_TOKEN");
HttpsURLConnection conn = (HttpsURLConnection) url2.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty(JWT_TOKEN_HEADER_PARAM, "Bearer" +token);
conn.setDoOutput(true);
OutputStream outStream = conn.getOutputStream();
OutputStreamWriter outStreamWriter = new OutputStreamWriter(outStream, "UTF-8");
outStreamWriter.write(list.toString());
outStreamWriter.flush();
outStreamWriter.close();
outStream.close();
String response = null;
System.out.println(conn.getResponseCode());
System.out.println(conn.getResponseMessage());
DataInputStream input1 = null;
input1 = new DataInputStream (conn.getInputStream());
while (null != ((response = input1.readLine()))) {
System.out.println(response);
input1.close ();
}
}
catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (ParseException e) {
e.printStackTrace();
}

If you are trying to get the JWT-token to authenticate the following java should work:
Request request = Request.Post("http://THINGSBOARD_URL/api/auth/login");
String body = "{\"username\":\"tenant#thingsboard.org\", \"password\":\"tenant\"}";
request.bodyString(body,ContentType.APPLICATION_JSON);
request.setHeader("Content-Type", "application/json");
request.setHeader("Accept", "application/json");
HttpResponse httpResponse = request.execute().returnResponse();
System.out.println(httpResponse.getStatusLine());
if (httpResponse.getEntity() != null) {
String html = EntityUtils.toString(httpResponse.getEntity());
System.out.println(html);
}
Don't get confused with JWT-Token for tenant authentication and Access-Token for Device Authentication.

How to Perform MalformedURLException / 404 Not Found

How do I make my program return MalformedUrlException and not just a generic Exception?
I am making a simple function that reads a URL a user enters in the console and it returns the content from the URL. I need it to check if the URL is a valid URL or if it's not a working URL.
Example urls:
http://google.com/not-found.html
http:/google.com
I created two catch exceptions but it seems like the overall exception is always returned instead of MalformedUrlException.
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
//wrap the url connection a buffered reader
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while((line = bufferedReader.readLine()) != null) {
content += line + "\n";
}
bufferedReader.close();
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}

First, java.net.MalformedURLException is not the case for a "not found" resource:
public class MalformedURLException extends IOException
Thrown to indicate that a malformed URL has occurred. Either no legal
protocol could be found in a specification string or the string could
not be parsed.
I understand that you want to catch a situation when the URL results in a not found return code (404). To do this you need to examine the HTTP response code.
The easiest way is to use java.net.HttpURLConnection:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/HttpURLConnection.html
public abstract class HttpURLConnection extends URLConnection
A URLConnection with support for HTTP-specific features. See the spec
for details.
Each HttpURLConnection instance is used to make a single request but
the underlying network connection to the HTTP server may be
transparently shared by other instances. Calling the close() methods
on the InputStream or OutputStream of an HttpURLConnection after a
request may free network resources associated with this instance but
has no effect on any shared persistent connection. Calling the
disconnect() method may close the underlying socket if a persistent
connection is otherwise idle at that time.
You can check the response code by calling getResponseCode(). If the result is less than 400, you got a valid response, otherwise there was a client error (4xx) or a server error (5xx).
Something like this:
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
if (urlConnection instanceof HttpURLConnection) {
HttpURLConnection conn = (HttpURLConnection) urlConnection;
if (conn.getResponseCode() < 400) {
// read contents
} else {
System.out.println(conn.getResponseMessage());
// treat the error as you like
}
} else {
// not a HTTP connection, treat as you like
}
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}
I have not checked the code, but I think you can get the overall idea.

NULL JSON Model passed to webservice when specific number of special characters

This is driving me mad - I have an object model in android that is turned into a JSON string using GsonBuilder:
WorkItemModel model = new WorkItemModel();
model.Jobs = jobList;
model.Items = itemList;
model.Images = imageList;
model.Questions = questionList;
String gsonJson = new GsonBuilder().create().toJson(model);
The string is passed to a c# webservice using this method :
private String callWebService(String JSONModel, URL url, int appId) throws Exception {
int connTimeout = 5000;
System.setProperty("http.keepAlive", "false");
HttpURLConnection conn = null;
try {
conn = (HttpURLConnection) url.openConnection();
//Populate Header
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("Content-Type", "application/json");
conn.setDoOutput(true);
conn.setChunkedStreamingMode(0);
conn.setConnectTimeout(connTimeout);
conn.setReadTimeout(connTimeout);
if (JSONModel != null) {
conn.setRequestProperty("Content-Length", String.valueOf(JSONModel.length()));
conn.setDoInput(true);
OutputStream stream = new BufferedOutputStream(conn.getOutputStream());
stream.write(JSONModel.getBytes());
stream.flush();
stream.close();
} else {
conn.setDoOutput(true);
conn.setChunkedStreamingMode(0);
conn.setRequestProperty("Content-Length", "0");
}
conn.connect();
BufferedReader inputStreamReader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
StringBuilder returnStringBuilder = new StringBuilder();
String streamString;
while ((streamString = inputStreamReader.readLine()) != null) {
returnStringBuilder.append(streamString);
}
inputStreamReader.close();
return returnStringBuilder.toString();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (UnknownHostException e) {
throw e;
} catch (SocketTimeoutException e) {
throw e;
} catch (SSLHandshakeException e) {
throw e;
} catch (IOException e) {
throw e;
} catch (Exception e) {
throw e;
} finally {
conn.disconnect();
}
return null;
}
The webservice should receive this model and then go away and do some back-end DB stuff - and for the most part, we don't have a problem, but we've had a few bug reports which I can replicate but can't seem to fix.
In the "Jobs" list in our model, we have a string property called completeNotes. When completeNotes is filled in with 4 "£" symbols, everything still works, but when there are 5 "£" symbols, the webservice Model it receives is NULL and I can see an exception has been thrown on the webservice :
Exception thrown: 'Newtonsoft.Json.JsonReaderException' in Newtonsoft.Json.dll
Additional information: Invalid character after parsing property name. Expected ':' but got: . Path 'Jobs', line 1, position 1585.
Just as a side note, this also happens with € symbols, except it only allows 2 € symbols.. on the third one it does the same thing - passes a NULL model through - the symbols don't have to be contiguous either
I won't show the whole JSON as its quite lengthy.. but the property I speak of is set like this in it :
"completeNotes":"£££££"
Has anyone got any ideas as to why this is happening - I've tried messing around with the encoding (setting it to UTF-8) but this doesn't seem to make a difference.

Have you tried adding "Charset=UTF-8' to the content type header?
conn.setRequestProperty("Content-Type", "application/json; Charset=UTF-8");

How to close a persistent HTTP Connection without reading

I have an URLConnection which I want to cancel depending on the response code without reading any data. I closely followed the android training to build the following minimal example which
floods the server with requests since no connection is ever released back to the handle pool for reuse
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d(TAG, "The response code is: " + response);
is = conn.getInputStream();
// Do not read anything //String contentAsString = readIt(is, len);
String contentAsString = "notReadingAnything";
return contentAsString;
} finally {
if (is != null) {
is.close();
}
}
}
private class DownloadWebpageTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
try {
String result = new String();
for (int i=0; i<100; i++) {
result += downloadUrl(urls[0]);
}
return result;
} catch (IOException e) {
return "Unable to retrieve web page. URL may be invalid.";
}
}
#Override
protected void onPostExecute(String result) {
Log.d(TAG, "The response is: " + result);
}
}
Despite the docs explicitly stating
But if the response body is long and you are not interested in the rest of it after seeing the beginning, you can close the InputStream
the server quickly reaches its maximum number of connections (50) and goes to 99% workload if I don't read the stream but works fine if I do read it. What is my mistake?
EDIT: Failed solution attempts so far (thanks to #Blackbelt for most of them)
calling conn.disconnect() in the finally block
calling conn.disconnect() instead of is.close() in the finally block
Setting System.setProperty("http.keepAlive", "false"); before the first call
Setting conn.setRequestProperty("Connection", "Close"); before connecting
Setting "{enable_keep_alive", "no"} on the used backend server (Civetweb)

you should call disconnect() too. Accordingly to the documentation
Disconnect. Once the response body has been read, the
HttpURLConnection should be closed by calling disconnect().
Disconnecting releases the resources held by a connection so they may
be closed or reused.
InputStream is = null;
HttpURLConnection conn = null;
try {
URL url = new URL(myurl);
conn = (HttpURLConnection) url.openConnection();
} finally {
if (is != null) {
is.close();
}
if (conn != null) {
conn.disconnect();
}
}
if you still are experiencing issues, is also possible that the bug is backend side