We Keep Coding

Getting stuck in an infinite-loop while retrieving api result

Getting stuck in an infinite-loop while retrieving api result.
Initialy I am getting scan percentage for 2-3 times but after that I am not getting any response.
My java code:
int responseCode=200;
String responseText = "text";
while (!responseText.equalsIgnoreCase("100") && responseCode == 200) {
URL urlForGetRequest = new URL("http://localhost:8090/burp/scanner/status");
String readLine = null;
HttpURLConnection conection = (HttpURLConnection) urlForGetRequest.openConnection();
conection.setRequestMethod("GET");
responseCode = conection.getResponseCode();
System.out.println("response" + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conection.getInputStream()));
String response = "";
while ((readLine = in.readLine()) != null) {
response += (readLine);
}
in.close();
JSONObject jsonObj = new JSONObject(response);
// print result
responseText = String.valueOf(jsonObj.get("scanPercentage"));
System.out.println(responseText);
TimeUnit.MINUTES.sleep(2);
}
output I got
response200
0
response200
4
response200
14
response200
17
and after this code kept on running without any output
Note: I perform get from its swagger UI,there is 1 error. i.e TypeError: Failed to fetch

There may be a chance the issue is in the response text check the actual text what the sender is sending
or just eliminate the extra space from the response text using following oneline code before processing the response text. It may be work for you if such problem is there..
responseText=responseText.replace(/^\s+|\s+$/g, '');

Put the code in try, catch block and check what exception it's throwing.

You could try to check if it gets stuck in the while loop, by adding an output and see if it keeps writing when it gets stuck.
while ((readLine = in.readLine()) != null) {
response += (readLine);
System.out.print(".");
}
I don't think this is the reason but it is worth a try.

How to binary data from a get response using jersey client?

I am trying to make a rest Get call using jersey client. Base on the api docs, the request returns an image as binary data. When I make the rest Get call using postman, I can the actual image back (im asumming postman converts the binary back to image/png). This is the following headers that is returned from postman.
I try making the rest Get using jersey client in java. Here is my code:
private Client client = ClientBuilder.newClient( new ClientConfig().register(LoggingFilter.class).register(MultiPartFeature.class));
private WebTarget myServer;
myServer= client.target(baseURL);
public void restGetImage(String requestURL, String headers) {
BufferedReader br = null;
StringBuilder sb = new StringBuilder();
String line;
MultivaluedMap<String, Object> userHeaders = storeHeadersInMap(headers);
WebTarget target = getWebTarget().path(requestURL);
Response response = target.request(MediaType.APPLICATION_OCTET_STREAM)
.headers(userHeaders)
.get();
System.out.println("Reuqest URL: " + session.get("baseurl") + requestURL);
int responseCode = response.getStatus();
InputStream inputStream = response.readEntity(InputStream.class);
String contentType = response.getHeaderString("Content-Type");
// br = new BufferedReader(new InputStreamReader(inputStream));
//
// try {
// while((line = br.readLine()) != null) {
// sb.append(line);
// }
//
// br.close(); //close buffered reader
//
// } catch (IOException e) {
// e.printStackTrace();
//
// }
//
// RestResponse data = new RestResponse(responseCode, sb.toString(), contentType);
// data.setResponseHeader(response.getHeaders());
//
// System.out.println("response code: " + responseCode);
// System.out.println("response Content-Type: " + contentType);
// System.out.println("Response body: " + sb.toString());
// return data;
}
Please excuse all the comments because I am still testing the code. Basically the code returns 200 response status, but when it fails to read the response. It throws an exception at InputStream inputStream = response.readEntity(InputStrean.class).
Apr 22, 2016 11:37:29 AM
org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderIn
terceptor aroundReadFrom
SEVERE: MessageBodyReader not found for media type=image/png, type=class
com.itextpdf.text.pdf.codec.Base64$InputStream, genericType=class
com.itextpdf.text.pdf.codec.Base64$InputStream.
My goal is to be able to read the binary response data. Any insight is apprectiated. Thanks.

SEVERE: MessageBodyReader not found for media type=image/png, type=class
com.itextpdf.text.pdf.codec.Base64$InputStream, genericType=class
com.itextpdf.text.pdf.codec.Base64$InputStream.
You're using the wrong InputStream class. You should be using java.io.InputStream. Check and fix your import.

java http apache client does not receive the content

I am using Apache client libs for http client to post some data to my server.
Below is the code, I get the status line response, but I am not getting the contents.
But on wireshark I could see server is responding the with few contents, such content type, location etc. for my code I am getting the following output.
Status :: HTTP/1.1 201 Created Content null
Pls help me to find out where I have gone wrong to read the content , do I need to some proxy related settings ?
HttpClient client = new DefaultHttpClient();
String line = ""; String status = "";
HttpPost post = new HttpPost("http://127.0.0.1/msg");
try {
HttpEntity e = new StringEntity("POSTING TO SERVER FOR TESTING");
post.setEntity(e);
HttpResponse response = client.execute(post);
BufferedReader rd = new BufferedReader(new
InputStreamReader(response.getEntity().getContent()));
status = response.getStatusLine().toString() ;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(" Status :: "+ status + " Content " + line);

The reason for Status :: HTTP/1.1 201 Created Content null is because the last value assigned to line is null (it's how you broke out of the loop). The response may not have any content. You can use the API to examine the headers as you need
Take a look at How to get headers? (java,httpclient 4.X) for an example

uploading files to a dataset in CKAN / datahub.io through a Java client

I am testing the uploading of files to a dataset on CKAN / datahub.io through a Java client of the API.
public String uploadFile()
throws CKANException {
String returned_json = this._connection.MultiPartPost("", "");
System.out.println("r: " + returned_json);
return returned_json;
}
and
protected String MultiPartPost(String path, String data)
throws CKANException {
URL url = null;
try {
url = new URL(this.m_host + ":" + this.m_port + path);
} catch (MalformedURLException mue) {
System.err.println(mue);
return null;
}
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
String fileName = "D:\\test.jpg";
FileBody bin = new FileBody(new File(fileName),"image/jpeg");
StringBody comment = new StringBody("Filename: " + fileName);
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("bin", bin);
reqEntity.addPart("comment", comment);
HttpPost postRequest = new HttpPost("http://datahub.io/api/storage/auth/form/2013-01-24T130158/test.jpg");
postRequest.setEntity(reqEntity);
postRequest.setHeader("X-CKAN-API-Key", this._apikey);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("status code: " + statusCode);
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
System.out.println("body: " + body);
} catch (IOException ioe) {
System.out.println(ioe);
} finally {
httpclient.getConnectionManager().shutdown();
}
return body;
}
2 responses I get to my POST request:
a 413 error ("request entity too large") when the jpeg I try to upload is 2.83 Mb. This disappears when I shrink the file to a smaller size. Is there a limit to file size uploads?
a 500 error ("internal server error"). This is where I am stuck. It might have to do with the fact that my dataset on datahub.io is not "datastore enabled"? (I see a disabled "Data API" button next to my resource files in the dataset, with a tooltip saying:
"Data API is unavailable for this resource as DataStore is disabled"
=> is it a possible reason for this 500 error? If so, how could I enable it from the client side? (pointers to Python code would be useful!)
Thx!
PS: the dataset I am using for testing purposes: http://datahub.io/dataset/testapi

Only someone with access to the exception log could tell you why the 500 is occurring.
However, I'd check your request is the same as what you'd get from the python client that was written alongside the datastore: https://github.com/okfn/ckanclient/blob/master/ckanclient/init.py#L546
You're sending the "bin" image buffer and "comment" file_key in your multipart request. Note the file_key must be changed for every upload, so add in a timestamp or something. And maybe you need to add in a Content-Type: for the binary.

I have been going through the same kind of troubles as the poster of this question. After quite a bit of trial and error, I came up with a solution to the problem. In my case, I had some control over the CKAN repository that I wanted to upload to. If you don't, your problem might be impossible to solve...
I assume you are using the 1.8 version of CKAN?
First of all, check whether the CKAN repository has been set up to allow file upload and if not, configure it to allow that. This can be done on the server using the steps posted here: http://docs.ckan.org/en/ckan-1.8/filestore.html#local-file-storage
The 413 error that you mentioned should be adressed next. This has to do with the general configuration of the server. In my case, the CKAN was hosted through nginx. I added a "client_max_body_size 100M" line to the nginx.conf file. See this post for instance: http://recursive-design.com/blog/2009/11/18/nginx-error-413-request-entity-too-large/
Then there is only the 500 error left. At the time of this writing, the api documentation of CKAN is still a little immature... It does indeed say that you have to build a request like you have made for file upload. However, this request is just to ask for permission for the file upload. If your credentials check out for file upload (not every user may be allowed to upload files), the response holds an object telling you where to send your file to... Because of the unclear api, you ended up merging these two requests.
The following scenario shows a follow up of two requests to handle the file upload. It might be that some steps in the scenario work out differently in your case, because of a repository that has been set up a little differently. If you get error messages, please be sure to check the response's body for clues!
Here is the authentication request that I used:
String body = "";
String generatedFilename=null;
HttpClient httpclient = new DefaultHttpClient();
try {
// create new identifier for every file, use time
SimpleDateFormat dateFormatGmt = new SimpleDateFormat("yyyyMMMddHHmmss");
dateFormatGmt.setTimeZone(TimeZone.getTimeZone("GMT"));
String date=dateFormatGmt.format(new Date());
generatedFilename=date +"/"+filename;
HttpGet getRequest = new HttpGet(this.CKANrepos+ "/api/storage/auth/form/"+generatedFilename);
getRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(getRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
throw new IllegalStateException("File reservation failed, server responded with code: "+statusCode+
"\n\nThe message was: "+body);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
Now, if all goes well, the server responds with a json object holding the parameters to use when doing the actual file upload. In my case, the object looked like:
{file_key:"some-filename-to-use-when-uploading"}
Be sure to check the json object though, as I'm given to understand that there may be custom ckan repositories that require more or different parameters.
These responses can then be used in the actual file upload:
File file = new File("/tmp/file.rdf");
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
FileBody bin = new FileBody(file,"application/rdf+xml");
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("file", bin);
reqEntity.addPart("key", new StringBody(filename));
HttpPost postRequest = new HttpPost(this.CKANrepos+"/storage/upload_handle");
postRequest.setEntity(reqEntity);
postRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
getWindow().showNotification("Upload Statuscode: "+statusCode,
body,
Window.Notification.TYPE_ERROR_MESSAGE);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
as you can see, the file_key property has now been transformed into the simple 'key' property. I don't know why.
This will get your file uploaded. The response to this upload request will hold a json object telling you where the file got uploaded to. edit: actually it seems that my ckan responded with a simple html page to tell me that the file got uploaded... I had to parse the page to confirm that the file was uploaded correctly :(
In my case, the file was at
this.CKANrepos +"/storage/f/"+location
where location is the filename returned in the authentication phase.
In the previous code fragments:
//the location of your ckan repository, including /api and possibly version, e.g.
this.CKANrepos = "http://datahub.io/api/3/";
this.CKANapiHeader="X-CKAN-API-Key";
this.CKANapi = "your ckan api key here";

What's the recommended way to get the HTTP response as a String when using Apache's HTTP Client? [duplicate]

This question already has answers here:
How to get HttpClient returning status code and response body?
(5 answers)
Closed 2 years ago.
I've just begun using Apache's HTTP Client library and noticed that there wasn't a built-in method of getting the HTTP response as a String. I'm just looking to get it as as String so that i can pass it to whatever parsing library I'm using.
What's the recommended way of getting the HTTP response as a String? Here's my code to make the request:
public String doGet(String strUrl, List<NameValuePair> lstParams) {
String strResponse = null;
try {
HttpGet htpGet = new HttpGet(strUrl);
htpGet.setEntity(new UrlEncodedFormEntity(lstParams));
DefaultHttpClient dhcClient = new DefaultHttpClient();
PersistentCookieStore pscStore = new PersistentCookieStore(this);
dhcClient.setCookieStore(pscStore);
HttpResponse resResponse = dhcClient.execute(htpGet);
//strResponse = getResponse(resResponse);
} catch (ClientProtocolException e) {
throw e;
} catch (IOException e) {
throw e;
}
return strResponse;
}

You can use EntityUtils#toString() for this.
// ...
HttpResponse response = client.execute(get);
String responseAsString = EntityUtils.toString(response.getEntity());
// ...

You need to consume the response body and get the response:
BufferedReader br = new BufferedReader(new InputStreamReader(httpresponse.getEntity().getContent()));
And then read it:
String readLine;
String responseBody = "";
while (((readLine = br.readLine()) != null)) {
responseBody += "\n" + readLine;
}
The responseBody now contains your response as string.
(Don't forget to close the BufferedReader in the end: br.close())

You can do something like:
Reader in = new BufferedReader(
new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
Using the reader you will be able to build your string. But if you are using SAX you can give the stream to the parser directly. This way you will not have to create the string and your memory footprint will be lower too.

In terms of conciseness of code it might be using the Fluent API like this:
import org.apache.http.client.fluent.Request;
[...]
String result = Request.Get(uri).execute().returnContent().asString();
The documentation warns though that this approach is not ideal in terms of memory consumption.