I'm trying to use Yahoo Content Analysis using a file containing text as input. So every character and length is possible.
This code works with a simple text String (no special characters, short text) however when I use longer texts or special characters I get a Bad Request error (HTTP 400) sometimes with an error message like "no viable alternative at character '['" or without an error message.
I encode every request and HTTP Post shouldn't have any limit as to the length.
Does the Yahoo service place a limit on the length of the request and/or are there any characters that it can't handle?
Any help to help this work is appreciated!
Here's my code (using commons-httpclient):
String fileInput = FileUtils.readFileToString(f);
StringBuilder builder = new StringBuilder();
builder.append("http://query.yahooapis.com/v1/public/yql?");
System.out.println(fileInput);
builder.append("q=")
.append(URLEncoder.encode("select * from contentanalysis.analyze where text='"+ fileInput +"'" , "UTF-8"))
.append("&format=json");
final String postUrl = builder.toString();
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(postUrl);
// Send POST request
int statusCode = client.executeMethod(method);
I think the problem is that while you are sending the request as an HTTP POST, the YQL query and text are all included in the URL. YQL does not really have a way for you to make HTTP POST requests directly, so I can think of a couple options:
Directly use the Content Analysis web service with an HTTP POST (docs)
Create a custom YQL data table which uses the <execute> tag to run custom JavaScript which could do the POST (example)
Of these options I think the former would be easier.
Related
I've been struggling for half a day trying to learn this and am stuck. My goal is to query finance data from yql yahoo finance tables. I have set up some code from an AndroidHive example and am able to get it running correctly for their sample query. But that sample query just grabs a JSON object directly from the main URL that they provide. To do this for yql, I need to convert the SQL query into a format that the httpClient will recognize, and my app keeps hanging and never returning a response.
First, I tried taking the exact query string from the yql to replicate their search, which for me was this:
https://query.yahooapis.com/v1/public/yql?q=select%20symbol%2CChange%20from%20yahoo.finance.quote%20where%20symbol%20in%20(%22SH%22%2C%22DOG%22%2C%22RWM%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=
That gives me a null result when I set url = to the above, and then run the following:
DefaultHttpClient httpClient = new DefaultHttpClient();
.
.
.
HttpGet httpGet = new HttpGet(url);
httpResponse = httpClient.execute(httpGet);
If I do this with the AndroidHive example URL in my code, it works fine.
If I enter the above URL in browser, it works fine. So clearly, my URL is not being entered correctly.
So then I read online that I need to use a URLEncode to convert syntax of URL into the correct format. Here's what I did:
private String url = "https://query.yahooapis.com/v1/public/yql?q=" +URLEncoder.encode("select symbol, Change from yahoo.finance.quote where symbol in (\"SH\",\"DOG\".\"RWM\")")+"&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=";
That gives me 2 problems. First, it tells me that encode is deprecated, and that I am supposed to use the new syntax that adds a "UTF-8" as a second string parameter, but when I do that, it is a compiler error of java.io.UnsupportedEncodingException (ugh), so then I just tried running it with the deprecated form, but that just leaves my app hanging forever like the original format.
I must be doing something really obvious incorrect here...
okay - updated.
HTTPS!!! versus HTTP. the sample query that i did for the AndroidHive example was HTTP and this yql query had HTTPS. if i simply change it to HTTP it works, and i didn't even need to do the URLEncoder business. The string directly from yql worked.
Now I need to start solving my JSON parsing bugs, but I've got my data!
I am developing an Android application in which I am trying to send a simple array as a URL parameter, but it is not working properly. I am using a HTTP client and GET method. I have tried this in the following way:
StringBuilder sb = new StringBuilder();
sb.append(URLEncoder.encode(e.getKey(), "UTF-8")).append('=').append(URLEncoder.encode(e.getValue()+"", "UTF-8"));
where e.getValue() is ArrayList<Integers>
My URL params are appended %5B28%5D when I am sending [28]. If I don't use URL encoder then it goes as [28] But I want to use URL encoder. Am I doing anything wrong?
Your code is fine. this is how URL encoding works.
Seems like there issue in server at the time of decoding.
Debug the server for any possible issue with decoding.
also refer this answer for a better way of sending an array in get request.
I want a java way to extract the parameters of a URL regardless the way these parameters are written in it, in the regular way like( https://www.facebook.com/Doly.mohamed.Smile9?ref=stream&hc_location=stream ) it's so easy because all i have to do is :
URL url = new URL("www.blabla....etc");
String query = url.getQuery();
try{
String [] params = query.split("&");
for(int i= 0 ; i < params.length; i++){
String [] split = params[i].split("=");
parameters.put(split[0], split[1]);
}
}catch(NullPointerException ex){}
so the parameters values would be :
key = ref value = stream , key = hc_location value = stream
but what shall i do if the URL has parameters written in another way or if the URL does't has it's parameters written in it like in the case of the doPost() way.
and is there is a way to get the extraPathInfo from a URL without using servlets?
You could do that easily with Apache's HTTP utils.
URIBuilder uriBuilder = new URIBuilder(uriString);
List<NameValuePair> urlParameters = uriBuilder.getQueryParams();
String uriWithoutParameters = uriBuilder.clearParameters().toString();
Now you could, for example, easily convert the GET request to a POST request, using other classes from the http utils API.
There is a difference between GET and POST urls
In GET url, parameters are part of URL and in POST they are part of Request-body.
So in POST, the URL may or may not contain the request params, and unless you don't have them in the URL its not possible to extract.
The POST request method is designed to request that a web server
accept the data enclosed in the request message's body for storage.1
It is often used when uploading a file or submitting a completed web
form.
So unless you have the POST request's body. Its difficult to extract the Parameter.
Typically you need HTTP request parameters on HTTP server side. Java HTTP server will parse the request and pass it as ServletRequest object to Servlet.service method. ServletRequest has methods to access the request parameters.
I have a method which takes a very long string, 90,860 characters and passes this to a PHP page which is then inserted into a database. I've read that there isn't a limit on the size of a PHP post upto about 8mb. I'm assuming that this length of string should be ok.
msg = URLEncoder.encode(stringToEncode.toString(),"UTF-8");
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("http://mysite.com/insert.php?data="+msg);
HttpResponse response1 = httpclient.execute(httpPost);
I know that this works as I can insert short plain strings.
On my PHP page I just get the post data like:
$data = $_GET['data'];
And insert it. Do I need to add some extra php to de-code the post message? I read that it should be done automatically by PHP.
The problem is that with the long string I either get a 500 error or if I take the string and put it in my browser I just get a page not available.
Thanks in advance!
In the Java program you should not pass your data, especially when it is long, in the URL, see this question for example code how to properly pass Post parameters
From PHP side, you should use $data = $_POST['data'];
The limiting factor will be the maximum allowed length of an URL that the server is willing to accept. I can't remember if there's some minimum all servers must support, but tests we did a while ago showed that depending on server type the URL length limit was commonly between 2k and 10k characters - this seems to be confirmed by Paul Dixon's extensive answer to this question.
To circumvent this limitation, use multipart post
I'm sending a RESTful JSON POST request using Apache HttpClient (to a 3rd party API)
Should I URL encode the JSON body?
And if something in the content is already URL encoded (e.g. I send HTML that has some links with URL encoded chars, e.g. #22) should I expect to get the content as is on the other side without it being decoded?
E.g. if i'm doing something like this
String html = "<a href='http://example.com?charOfTheDay=%22'>click me</a>";
// Build the JSON object
JSONObject jsonObj = new JSONObject();
jsonObj.put("html", html);
jsonObj.put("otherKey",otherValue);
//...
// Create the POST object and add the parameters
HttpPost httpPost = new HttpPost(url);
StringEntity entity = new StringEntity(jsonObj.toString(), HTTP.UTF_8);
entity.setContentType("application/json");
httpPost.setEntity(entity);
HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(httpPost);
Should I expect getting the same value on the receiving end, after getting the value of the "html" key?
e.g. on the receiving end
//after parsing the request string to a JSON object
String html = inputJsonObject.get("html")
// should return "<a href='http://example.com?charOfTheDay=%22'>click me</a>"
Are there any other steps I need to do to make sure what I send is what is received "as is"?
There's sort of two contexts you have to worry about here:
JSON
You have to make sure that the JSON you generate is valid JSON (duh). This means making sure that all the { } and [ ] syntax is in the right place and making sure that the field values you are inserting into the JSON object are safely escaped (like escaping that HTML snippet in your question --some of the quote characters would need to be escaped). BUT because you're using a standard JSON Java library, you don't have to worry about this...it will take care of all this for you.
HTTP
The JSON string then has to be inserted into the HTTP request body. No escaping needs to be done here--just insert the raw JSON string. HTTP, as a protocol, will accept anything inside the request/response bodies, including raw binary data.
Anyway that was kind of long, but I hope it helped.
The Content-Type in your http header should be application/json, so you oughtn't URL encode the body of the http request.
URL encoding is meant to prevent users from using characters that are special in representing URLs (such as '/').
You don't have to worry about links in the content being decoded either, unless you use a Content-Type in your http header that suggests that the server should decode the body, such as application/x-www-form-urlencoded