How to parse a large amount of data from a JSON webservice - java

I have a URL which returns a large amount of data in response, but I can't get a full response from the webservice URL. The webservice responds as follows:
\"},
{\"minute_usage_end_time\":\"11:59\",\"minute_usage_start_time\":\"11:00\",\"kwh_usage\":\"0\",\"meter_reading_date\":\"08-02-2011\"},{...
What should I do?

StringEntity entity = new StringEntity(jsonObject.toString(),HTTP.UTF_8);
httpPost.setEntity(entity);
HttpResponse response = httpClient.execute(httpPost);
BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
Log.v("mas\n", reader.readLine());
String jsonResultStr = reader.readLine();

Related

Code works in Xamarin Android but doestn't work in Java (HttpPost JSON)

I started developing in Xamarin, and then decided that license may be a bit expensive for playing around, so I'm transferring my code to java.
I have a small chunk that performs a POST with a JSON object, and it works in Xamarin and doest work in Java.
Xamarin:
var client = new HttpClient ();
var content = new FormUrlEncodedContent(new Dictionary<string, string>() {
{"action", "getEpisodeJSON"},
{"episode", "11813"}
});
client.DefaultRequestHeaders.Referrer = new Uri(link);
var resp = client.PostAsync("http://www.ts.kg/ajax", content).Result;
var repsStr = resp.Content.ReadAsStringAsync().Result;
dynamic res = JsonConvert.DeserializeObject (repsStr);
Android:
HttpClient httpclient = new DefaultHttpClient();
// 2. make POST request to the given URL
HttpPost httpPost = new HttpPost("http://www.ts.kg/ajax");
String json = "";
// 3. build jsonObject
JSONObject jsonObject = new JSONObject();
jsonObject.accumulate("action", "getEpisodeJSON");
jsonObject.accumulate("episode", "11813");
// 4. convert JSONObject to JSON to String
json = jsonObject.toString();
// 5. set json to StringEntity
StringEntity se = new StringEntity(json);
// 6. set httpPost Entity
httpPost.setEntity(se);
// 7. Set some headers to inform server about the type of the content
httpPost.setHeader("Accept", "application/json");
httpPost.setHeader("Content-type", "application/json");
httpPost.addHeader("Referer", "http://www.ts.kg");
// 8. Execute POST request to the given URL
HttpResponse httpResponse = httpclient.execute(httpPost);
// 9. receive response as inputStream
InputStream inputStream = httpResponse.getEntity().getContent();
// 10. convert inputstream to string
String result;
if(inputStream != null)
result = convertInputStreamToString(inputStream);
What is a correct way to make such a POST in Android?
UPD
Current problem is that i'm getting an empty result string;
private static String convertInputStreamToString(InputStream inputStream) throws IOException{
BufferedReader bufferedReader = new BufferedReader( new InputStreamReader(inputStream));
String line = "";
String result = "";
while((line = bufferedReader.readLine()) != null)
result += line;
inputStream.close();
return result;
}
I ended up catching all requests of my device via Fiddle (good tutorial is here: http://tech.vg.no/2014/06/04/how-to-monitor-http-traffic-from-your-android-phone-through-fiddler/)
The difference was in cookie, so I used and HttpContex variable as described here:
Android HttpClient Cookie
And I also had a different Content-Type, so I set this header manually as this:
httpPost.setHeader("Content-Type", "application/x-www-form-urlencoded");

Why am I getting HTTP 400 bad request

I am using an HTTP client (code copied from http://www.mkyong.com/java/apache-httpclient-examples/) to send post requests. I have been trying to use it with http://postcodes.io to look up a bulk of postcodes but failed. According to http://postcodes.io I should send a post request to http://api.postcodes.io/postcodes in the following JSON form: {"postcodes" : ["OX49 5NU", "M32 0JG", "NE30 1DP"]} but I am always getting HTTP Response Code 400.
I have included my code below. Please tell me what am I doing wrong?
Thanks
private void sendPost() throws Exception {
String url = "http://api.postcodes.io/postcodes";
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost(url);
List<NameValuePair> urlParameters = new ArrayList<NameValuePair>();
urlParameters.add(new BasicNameValuePair("postcodes", "[\"OX49 5NU\", \"M32 0JG\", \"NE30 1DP\"]"));
post.setEntity(new UrlEncodedFormEntity(urlParameters));
HttpResponse response = client.execute(post);
System.out.println("Response Code : "
+ response.getStatusLine().getStatusCode());
System.out.println("Reason : "
+ response.getStatusLine().getReasonPhrase());
BufferedReader br = new BufferedReader(
new InputStreamReader(response.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = br.readLine()) != null) {
result.append(line);
}
br.close();
System.out.println(result.toString());
}
This works, HTTP.UTF_8 is deprecated:
String url = "http://api.postcodes.io/postcodes";
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost(url);
StringEntity params =new StringEntity("{\"postcodes\" : [\"OX49 5NU\", \"M32 0JG\", \"NE30 1DP\"]}");
post.addHeader("Content-Type", "application/json");
post.setEntity(params);
Jon Skeet is right (as usual, I might add), you are basically sending a form and it defaults to form-url-encoding.
You could try something like this instead:
String jsonString = "{\"postcodes\" : [\"OX49 5NU\", \"M32 0JG\", \"NE30 1DP\"]}";
StringEntity entity = new StringEntity(jsonObj.toString(), HTTP.UTF_8);
entity.setContentType("application/json");
post.setEntity(entity);

HTTP request not returning desired json

I am trying to send a http request to a website which is supposed to return a json response. The problem is that i am not getting the json data. But when i paste the url in a browser it displays the json output. Am a newbie. Kindly help.
Here is my code
HttpClient client = new DefaultHttpClient();
String url="http://directclientvendors.com/news24/api/get.php?type=news";
HttpGet request = new HttpGet(url);
HttpResponse response;
response = client.execute(request);
BufferedReader br =
new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
String line = "";
while(br.ready())
{
line+=br.readLine();
}
System.out.println("line "+line);
You should be executing a GET request and not a POST. Please change the request type to HttpGet. The browser executes a GET on the URL when you paste it on the address bar and hit enter.
Additionally use a Reader + StringBuilder / JsonReader / GSON to read from the URL's response content. String concatenation leads to the creation of additional objects unnecessarily.
[EDIT]
To my astonishment the API call works even when a POST call is made to get the resource. The problem must be in your parsing logic. Using a JsonReader works fine for me. This is just template code, but you can fill in the rest to get the other JSON elements. Regardless of whether POST works or not, you should still use GET for this call.
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet("http://directclientvendors.com/news24/api/get.php?type=news");
HttpResponse response = client.execute(request);
InputStream content = response.getEntity().getContent();
JsonReader jsonReader = new JsonReader(new InputStreamReader(content, "UTF-8"));
jsonReader.beginObject();
if(jsonReader.hasNext())
{
System.out.println(jsonReader.nextName()); // prints 'news'
// BEGIN_ARRAY etc to parse the rest
}
// END_OBJECT and cleanup

Displaying NON-ASCII Characters using HttpClient

So, i am using this code to get the whole HTML of a website. But i dont seem to get non-ascii characters with me. all i get is diamonds with question mark.
characters like this: å, appears like this: �
I doubt its because of the charset, what could it then be?
Log.e("HTML", "henter htmlen..");
String url = "http://beep.tv2.dk";
HttpClient client = new DefaultHttpClient();
client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
HttpVersion.HTTP_1_1);
client.getParams().setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET, "UTF-8");
HttpGet request = new HttpGet(url);
HttpResponse response = client.execute(request);
Header h = HeaderValueFormatter
response.addHeader(header)
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while((line = reader.readLine()) != null)
{
str.append(line);
}
in.close();
//b = false;
html = str.toString();
Thank you. This worked (in case others have the issue):
HttpClient client = new DefaultHttpClient();
client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
HttpVersion.HTTP_1_1);
client.getParams().setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET, "iso-8859-1");
HttpGet request = new HttpGet(url);
request.setHeader("Accept-Charset", "iso-8859-1, unicode-1-1;q=0.8");
HttpResponse response = client.execute(request);
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in,"iso-8859-1"));
use the new InputStreamReader(in, "UTF-8") constructor
Set the Accept-Charset request header to, say, Accept-Charset: iso-8859-5, unicode-1-1;q=0.8
Make sure the page opens properly in a browser. If it does not, then it might be a server-side issue.
If none of the above works, check other headers using firebug (or similar tool)
This really helped me get started, but I was having the same problem while reading a text file. It was fixed using the following command:
BufferedReader br = new BufferedReader(new InputStreamReader(new
FileInputStream(fileName), "iso-8859-1"));
...and of course, the HTTP Response needs to have the encoding set as well:
response.setCharacterEncoding("UTF-8");
Thanks for the help!

Java client has trouble to read big responses from wcf rest service

I am trying to consume a rest WCF service using java client on android 2.1.
It works perfectly on small responses. When i tried to push a little further by getting 1000+ char response reader.read(buffer) failed to read all the characters. This caused the last line of the script to return: JsonException unterminated string at character 8193 "{plate,...
My android device starts to get this error before the emulator android stars to get it (character 1194 instead of 8193). Anyone knows how get the full message?
Client Code:
HttpGet request = new HttpGet(SERVICE_URI + "/GetPlates");
request.setHeader("Accept", "application/json");
request.setHeader("Content-type", "application/json");
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpResponse response = httpClient.execute(request);
HttpEntity responseEntity = response.getEntity();
char[] buffer = new char[(int)responseEntity.getContentLength()];
InputStream stream = responseEntity.getContent();
InputStreamReader reader = new InputStreamReader(stream);
reader.read(buffer);
stream.close();
JSONArray plates = new JSONArray(new String(buffer));
Server Config:
<webHttpBinding>
<binding name="big_webHttpBinding" maxReceivedMessageSize="4097152" maxBufferSize="4097152" maxBufferPoolSize="4097152">
<readerQuotas maxStringContentLength="4097152" maxArrayLength="4097152" maxBytesPerRead="4097152" maxNameTableCharCount="4097152" maxDepth="4097152"/>
</binding>
</webHttpBinding>
Instead of
char[] buffer = new char[(int)responseEntity.getContentLength()];
InputStream stream = responseEntity.getContent();
InputStreamReader reader = new InputStreamReader(stream);
reader.read(buffer);
stream.close();
do this:
String jsonText = EntityUtils.toString(responseEntity, HTTP.UTF_8);

Categories