I am trying to use Google language detection API, Right now I am using the sample available on Google documentation as follows:
public static String googleLangDetection(String str) throws IOException, JSONException{
String urlStr = "http://ajax.googleapis.com/ajax/services/language/detect?v=1.0&q=";
// String urlStr = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Paris%20Hilton";
URL url = new URL(urlStr+str);
URLConnection connection = url.openConnection();
// connection.addRequestProperty("Referer","http://www.hpeprint.com");
String line;
StringBuilder builder = new StringBuilder();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while((line = reader.readLine()) != null) {
builder.append(line);
}
JSONObject json = new JSONObject(builder.toString());
for (Iterator iterator = json.keys(); iterator.hasNext();) {
String type = (String) iterator.next();
System.out.println(type);
}
return json.getString("language");
}
But I am getting http error code '406'.
I am unable to understand what the problem is? As the google search query(commented) below it is working fine.
The resultant language detection url itself is working fine when I run it in firefox or IE but it's failing in my java code.
Is there something I am doing wrong?
Thanks in advance
Ashish
As a guess, whatever is being passed in on str has characters that are invalid in a URL, as the error code 406 is Not Acceptable, and looks to be returned when there is a content encoding issue.
After a quick google, it looks like you need to run your str through the java.net.URLEncoder class, then append it to the URL.
Found the answer at following link:
HTTP URL Address Encoding in Java
Had to modify the code as follows:
URI uri = new URI("http","ajax.googleapis.com","/ajax/services/language/detect","v=1.0&q="+str,null);
URL url = uri.toURL();
Related
I am writing a Java class to access a third-party public REST API web service, that is secured using a specific APIKey parameter.
I can access the required Json array, using the JsonNode API when I save the json output locally to a file.
E.g.
JsonNode root = mapper.readTree(new File("/home/op/Test/jsondata/loans.json"));
But, if I try to use the live secured web URL with JsonNode
E.g.
JsonNode root = mapper.readTree(url);
I am getting a:
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60))
which suggests that I have a type mismatch. But I am assuming that it is more likely to be a connection issue.
I am handling the connection to the REST service:
private static String surl = "https://api.rest.service.com/xxxx/v1/users/xxxxx/loans?apikey=xxxx"
public static void main(String[] args) {
try {
URL url = new URL(surl);
JsonNode root = mapper.readTree(url);
....
}
I have also tried to use:
URL url = new URL(surl);
HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
InputStream isr = httpcon.getInputStream();
JsonNode root = mapper.readTree(isr);
with the same result.
When I remove the APIKey I receive a Status 400 Error. So I think I must not be handling the APIKey parameter.
Is there way to handle the call to the secured REST service URL using JsonNode? I would like to continue to use the JsonNode API, as I am extracting just two key:value pairs traversing multiple objects in a large array.
Just try to simply read response into string and log it to see what's actually going on and why you do not receive JSON from server.
URL url = new URL(surl);
HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
InputStream isr = httpcon.getInputStream();
try (BufferedReader bw = new BufferedReader(new InputStreamReader(isr, "utf-8"))) {
StringBuilder sb = new StringBuilder();
String line;
while ((line = bw.readLine()) != null) { // read whole response
sb.append(line);
}
System.out.println(sb); //Output whole response into console or use logger of your choice instead of System.out.println
}
I have a dialogflow project that I'm trying to access from Java with a rest call.
It is giving me an authentication issue.
I have followed all online instructions (and many forum suggestions) to no avail.
I have tried generating the key json, as per the instructions here:
https://dialogflow.com/docs/reference/v2-auth-setup
and setting my environment variable as described, but nothing seems to work.
I have checked my projectID, and am running off the same machine with the environment variable, and have double, triple and quadruple checked it's name and location, but I still get the following error:
java.net.HttpRetryException: cannot retry due to server authentication, in streaming mode
Here is my code (though it's a REST call, so I don't know if it's so relevant):
String url = https://dialogflow.googleapis.com/v2/projects/MYPROJECT/agent/sessions/SESSION_NUM:detectIntent
URL url = new URL(full_url);
String inText = "Hello World";
String outText = "";
HttpURLConnection con = (HttpURLConnection) url.openConnection();
try {
con.setDoOutput(true);
con.setRequestMethod("POST");
// set body of http post
Map<String,String> arguments = new HashMap<>();
JSONObject inTextJsn = new JSONObject();
inTextJsn.append("text",inText);
inTextJsn.append("languageCode","en");
JSONObject fieldJsn = new JSONObject();
fieldJsn.append("text", inTextJsn);
arguments.put("queryInput", fieldJsn.toString());
StringJoiner sj = new StringJoiner("&");
for(Map.Entry<String,String> entry : arguments.entrySet())
sj.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "="
+ URLEncoder.encode(entry.getValue(), "UTF-8"));
// post http post as bytes
byte[] bytes_out = sj.toString().getBytes(StandardCharsets.UTF_8);
con.setFixedLengthStreamingMode(bytes_out.length);
con.connect();
try (OutputStream os = con.getOutputStream()) {
os.write(bytes_out);
}
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream(),
"UTF-8"));
// read all lines to a string
String line;
String response = "";
while ((line = reader.readLine()) != null) {
response += line;
}
JSONObject responseJsn = new JSONObject(response);
outText = responseJsn.get("fulfillmentText").toString();
} catch (Exception e) {
System.err.println(e);
} finally {
con.disconnect();
}
return restResponse;
The gist of the code is to simply send a message ("Hello World!") to my dialogflow, and get back my agent's response (the code may have bugs - it's a bit hard to test when I can't get passed this authentication issue, so please help with the authentication, not code bugs).
Thanks all!
The directions at that page assume you're going to use the gcloud program to generate a currently valid bearer token, which is then sent along with the HTTP headers. That page illustrates
Your code doesn't seem to be generating an Authorization HTTP header at all, which is why you're getting the error you do.
Since you're using Java, you should look at the google-auth-library-java library, which will give you the tools to generate the token you need to provide in the Authorization header.
You may also wish to check out the google-cloud-java library. This contains Java classes to directly perform operations against Dialogflow instead of coding the REST/HTTP calls yourself. (However, it is still at an Alpha level for Dialogflow, so may not be stable or forwards compatible.)
I use java to get youtube video url from a youtube url,it seems like i should decode video url twice to get the original video url,but something is wrong when i access the video url which i get from my code. Im sure i can access any video url which i get when i first run the code after i finish the code, but after that ,i can not access any video url i get no matter the video url is from decode first or twice. I suppose maybe only youtube itself can access the video url i get or the server blocked my IP. So i need you guys to help me to tell me whats wrong and how i could do to get the video url that can play, thx. My code and answer is below:
try {
String ytUrl = "https://www.youtube.com/watch?v=Cj3AV92fJ90";
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(ytUrl);
HttpResponse response = client.execute(request);
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
str.append(line.replace("\\u0026", "&"));
}
in.close();
html = str.toString();
String val=RegexUtil.find(html,"stream_map\":.*?\"(.*?)\"",1);
if (!StringUtils.isEmpty(val)) {
String url = URLDecoder.decode(val, "UTF-8");
System.out.println("1 decode url: "+url);
url = URLDecoder.decode(url, "UTF-8");;
System.out.println("2 decode url: "+url );
}
} catch (Exception e) {
e.printStackTrace();
}
1 decode url: url=https://r4---sn-i3b7knlk.googlevideo.com/videoplayback?id=o-AE4MBDOnLdC4X0a7wjJ63diBNkBpJ2XiuejUOXrB8onv&dur=421.442&mime=video%2Fmp4&fvip=4&requiressl=yes&ms=au%2Conr&mt=1540798558&ratebypass=yes&itag=22&pl=25&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&source=youtube&mv=m&mn=sn-i3b7knlk%2Csn-npoeene7&lmt=1537570091802082&key=yt6&ei=8LjWW9P1Ns2R8gOGhKO4AQ&c=WEB&expire=1540820305&ip=114.113.240.105&ipbits=0&initcwndbps=362500&mm=31%2C26&itag=22&type=video/mp4; codecs="avc1.64001F, mp4a.40.2"&s=C8C80C74C92B4306498EE1183D683A0E60962F69BD.4F1AE9CE009B83425F389362CFC2FE23A45948D5&quality=hd720&sp=signature
2 decode url: url=https://r4---sn-i3b7knlk.googlevideo.com/videoplayback?id=o-AE4MBDOnLdC4X0a7wjJ63diBNkBpJ2XiuejUOXrB8onv&dur=421.442&mime=video/mp4&fvip=4&requiressl=yes&ms=au,onr&mt=1540798558&ratebypass=yes&itag=22&pl=25&sparams=dur,ei,id,initcwndbps,ip,ipbits,itag,lmt,mime,mm,mn,ms,mv,pl,ratebypass,requiressl,source,expire&source=youtube&mv=m&mn=sn-i3b7knlk,sn-npoeene7&lmt=1537570091802082&key=yt6&ei=8LjWW9P1Ns2R8gOGhKO4AQ&c=WEB&expire=1540820305&ip=114.113.240.105&ipbits=0&initcwndbps=362500&mm=31,26&itag=22&type=video/mp4; codecs="avc1.64001F, mp4a.40.2"&s=C8C80C74C92B4306498EE1183D683A0E60962F69BD.4F1AE9CE009B83425F389362CFC2FE23A45948D5&quality=hd720&sp=signature
I have devices that publish an html page when you connect via their ip address. For example, if I were to go to "192.168.1.104" on my computer, i would see the html page the device publishes. I am trying to scrape this html, but I am getting some errors, specifically a MalformedURLException at the first line of my method. I have posted my method below. I found some code for getting html and tweaked it for my needs. Thanks
public String getSbuHtml(String ipToPoll) throws IOException, SocketTimeoutException {
URL url = new URL("http", ipToPoll, -1, "/");
URLConnection con = url.openConnection();
con.setConnectTimeout(1000);
con.setReadTimeout(1000);
Pattern p = Pattern.compile("text/html;\\s+charset=([^\\s]+)\\s*");
Matcher m = p.matcher(con.getContentType());
String charset = m.matches() ? m.group(1) : "ISO-8859-1";
BufferedReader r = new BufferedReader(
new InputStreamReader(con.getInputStream(), charset));
String line = null;
StringBuilder buf = new StringBuilder();
while ((line = r.readLine()) != null) {
buf.append(line).append(System.getProperty("line.separator"));
}
return buf.toString();
}
EDIT: The above code has been changed to reflect constructing a new URL to work properly with an ip. However, when I try and get the contentType from the connection, it is null.
A URL (Uniform Resource Locator) must have a resource to locate (index.html) along with the means of network communication (http://). So an example of valid URL can be
http://192.168.1.104:8080/app/index.html
Merely 192.168.1.104 doesn't represent a URL
You need to add http:// to the front of your String that you pass into the method.
Create your URL as follows:
URL url = new URL("http", ipToPoll, -1, "/");
And since you're reading a potentially long HTML page I suppose buffering would help here:
BufferedReader r = new BufferedReader(
new InputStreamReader(con.getInputStream(), charset));
String line = null;
StringBuilder buf = new StringBuilder();
while ((line = r.readLine()) !- null) {
buf.append(line).append(System.getProperty("line.separator"));
}
return buf.toString();
EDIT: In response to your contentType coming null problem.
Before you inspect any headers like with getContentType() or retrieve content with getInputStream() you need to actually establish a connection with the URL resource by calling
URL url = new URL("http", ipToPoll, "/"); // -1 removed; assuming port = 80 always
// check your device html page address; change "/" to "/index.html" if required
URLConnection con = url.openConnection();
// set connection properties
con.setConnectTimeout(1000);
con.setReadTimeout(1000);
// establish connection
con.connect();
// get "content-type" header
Pattern p = Pattern.compile("text/html;\\s+charset=([^\\s]+)\\s*");
Matcher m = p.matcher(con.getContentType());
When you call openConnection() first (it wrongly suggests but) it doesn't establish any connection. It just gives you an instance of URLConnection to let you specify connection properties like connection timeout with setConnecTimeout().
If you're finding this hard to understand it may help to know that it's analogous to doing a new File() which simply represents a File but doesn't create one (assuming it doesn't exist already) unless you go ahead and call File.createNewFile() (or pass it to a FileReader).
I've followed a tutorial and came up with the following method to read the webpage content into a CharSequence
public static CharSequence getURLContent(URL url) throws IOException {
URLConnection conn = url.openConnection();
String encoding = conn.getContentEncoding();
if (encoding == null) {
encoding = "ISO-8859-1";
}
BufferedReader br = new BufferedReader(new
InputStreamReader(conn.getInputStream(),encoding));
StringBuilder sb = new StringBuilder(16384);
try {
String line;
while ((line = br.readLine()) != null) {
sb.append(line);
sb.append('\n');
}
} finally {
br.close();
}
return sb;
}
It will return a representation of the webpage specified by the url.
However,this representation is hugely different from what I use "view page source" in my Firefox,and since I need to scrape data from the original webpage(some data segement in the original "view page source" file),it will always fail to find required text on this Java representation.
Did I go wrong somewhere?I need your advice guys,thanks a lot for helping!
You need to use an HTML-parsing library to build a data structure representing the HTML text on this webpage. My recommendation is to use this library: http://htmlparser.sourceforge.net.
Things like the request useragent and cookies can change what the server returns in the response. So the problem is more likely in the details of the request you are sending rather than in how you are reading the response.
Things like HttpClient will allow you to more easily simulate the request being sent from a browser.