I am trying to hit the URL and get the response from my Java code.
I am using URLConnection to get this response. And writing this response in html file.
When opening this html in browser after executing the java class, I am getting only google home page and not with the results.
Whats wrong with my code, my code here,
FileWriter fWriter = null;
BufferedWriter writer = null;
URL url = new URL("https://www.google.co.in/?gfe_rd=cr&ei=aS-BVpPGDOiK8Qea4aKIAw&gws_rd=ssl#q=google+post+request+from+java");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String(encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("User-Agent", "Mozilla/5.0");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setDoInput(true);
connection.setRequestProperty("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream) connection.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String line;
try {
fWriter = new FileWriter(new File("f:\\fileName.html"));
writer = new BufferedWriter(fWriter);
while ((line = in.readLine()) != null) {
String s = line.toString();
writer.write(s);
}
writer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
Same code works couple of days back, but not now.
The reason is that this url does not return search results it self. You have to understand google's working process to understand it. Open this url in your browser and view its source. You will only see lots of javascript there.
Actually, in a short summary, google uses Ajax requests to process search queries.
To perform required task you either have to use a headless browser (the hard way) which can execute javascript/ajax OR better use google search api as directed by anand.
This method of searching is not advised is supposed to fail, you must use google search APIs for this kind of work.
Note: Google uses some redirection and uses token, so even if you will find a clever way to handle it, it is ought to fail in long run.
Edit:
This is a sample of how using Google search APIs you can get your work done in reliable way; please do refer to the source for more information.
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}
Related
My goal is to get the xml from an API. The API uri I use, including parameters is http://webservices.ns.nl/ns-api-treinplanner?fromStation=Roosendaal&toStation=Eindhoven. I am given a username and password, for what I think probably is basic authorization.
I tried various things like something with an Authenticator, the format http://username:password#webservices.ns.nl/ns-api-treinplanner, but at the end of a lot of SO searching I ended up with something with a setRequestProperty with the basic authorization.
I put the code into an AsyncTask which seems to work correctly so I will just put the code from inside doInBackground in here.
As the java FileNotFoundException I first got didn't give me much information, I found out how to use the getErrorStream to find out more.
InputStream in;
int resCode;
try {
URL url = new URL("http://webservices.ns.nl/ns-api-treinplanner?fromStation=Roosendaal&toStation=Eindhoven");
String userCredentials = "username:password";
String encoding = new String(android.util.Base64.encode(userCredentials.getBytes(), Base64.DEFAULT));
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestProperty("Authorization", "Basic " + encoding);
urlConnection.setRequestMethod("GET");
urlConnection.setDoOutput(true);
try {
resCode = urlConnection.getResponseCode();
if (resCode == HttpURLConnection.HTTP_OK) {
Log.i("rescode","ok");
in = urlConnection.getInputStream();
} else {
Log.i("rescode","not ok");
in = urlConnection.getErrorStream();
}
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(in));
StringBuilder stringBuilder = new StringBuilder();
String line;
while ((line = bufferedReader.readLine()) != null) {
stringBuilder.append(line).append("\n");
}
bufferedReader.close();
return stringBuilder.toString();
}
finally{
urlConnection.disconnect();
}
}
catch(Exception e) {
Log.e("ERROR", e.getMessage(), e);
return null;
}
Then, in onPostExecute I print the response, but the response I get is
<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
<soap:Header></soap:Header>
<soap:Body><soap:Fault>
<faultcode>soap:Server</faultcode>
<faultstring>006:No customer found for the specified username and password</faultstring></soap:Fault>
</soap:Body></soap:Envelope>
This is of course not right, it should give a full xml of in this case a train voyage recommendation.
I tested with my browsers, and also using a HTTP request tool called Postman which returned the correct xml so all the uri's, parameters, username and password are correct.
The encoding used is wrong. The base64 encoding used randomly returns whitespaces in the middle, adding encoding = encoding.replaceAll("\\s+",""); actually fixed it.
When using this code below to make a get request:
private String get(String inurl, Map headers, boolean followredirects) throws MalformedURLException, IOException {
URL url = new URL(inurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setInstanceFollowRedirects(followredirects);
// Add headers to request.
Iterator entries = headers.entrySet().iterator();
while (entries.hasNext()) {
Entry thisEntry = (Entry) entries.next();
Object key = thisEntry.getKey();
Object value = thisEntry.getValue();
connection.addRequestProperty((String)key, (String)value);
}
// Attempt to parse
InputStream stream = connection.getInputStream();
InputStreamReader isReader = new InputStreamReader(stream );
BufferedReader br = new BufferedReader(isReader );
System.out.println(br.readLine());
// Disconnect
connection.disconnect();
return connection.getHeaderField("Location");
}
The resulting response is completely nonsensical (e.g ���:ks�6��9�rђ� e��u�n�qש�v���"uI*�W��s)
However I can see in Wireshark that the response is HTML/XML and nothing like the string above. I've tried a myriad of different methods for parsing the InputStream but I get the same result each time.
Please note: this only happens when it's HTML/XML, plain HTML works.
Why is the response coming back in this format?
Thanks in advance!
=== SOLVED ===
Gah, got it!
The server is compressing the response when it contains XML, so I needed to use GZIPInputStream instead of InputSream.
GZIPInputStream stream = new GZIPInputStream(connection.getInputStream());
Thanks anyway!
use an UTF-8 encoding in input stream like below
InputStreamReader isReader = new InputStreamReader(stream, "UTF-8");
I have a java program which I want to input something into an html form. If possible it could just load a url like this
.../html_form_action.asp?kill=Kill+Server
But i'm not sure how to load a url in Java. How would I do this? Or is there a better way to send an action to an html form?
Depending on your security, you can make an HTTP call in Java. It is often referred to as a RESTFul call. The HttpURLConnection class offers encapsulation for basic GET/POST requests. There is also an HttpClient from Apache.
Here's how you can use URLConnection to send a simple HTTP request.
URL url = new URL(url + "?" + query);
// set connection properties
URLConnection connection = url.openConnection();
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.connect(); // send request
// read response
BufferedReader reader = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
reader.close(); // close connection
i'm looking for tutorial or quick example, how i can send POST data throw openStream.
My code is:
URL url = new URL("http://localhost:8080/test");
InputStream response = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response, "UTF-8"));
Could you help me ?
URL url = new URL(urlSpec);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(method);
connection.setDoOutput(true);
connection.setDoInput(true);
// important: get output stream before input stream
OutputStream out = connection.getOutputStream();
out.write(content);
out.close();
// now you can get input stream and read.
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
writer.println(line);
}
Use Apache HTTP Compoennts http://hc.apache.org/httpcomponents-client-ga/
tutorial: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
Look for HttpPost - there are some examples of sending dynamic data, text, files and form data.
Apache HTTP Components in particular, the Client would be the best way to go.
It absracts a lot of that nasty coding you would normally have to do by hand
What is the best way to use preemptive basic http authentication using HttpUrlConnection. (Assume for now I can't use HttpClient).
EDIT for clarification: I'm setting the un/pw correctly in the request header using Base64 encoding. Are there any additional flags or properties that need to be set, or is the fact that I'm setting the basic auth headers for the request all that is needed for preemptive basic auth?
If you are using Java 8 or later, java.util.Base64 is usable:
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
String encoded = Base64.getEncoder().encodeToString((username+":"+password).getBytes(StandardCharsets.UTF_8)); //Java 8
connection.setRequestProperty("Authorization", "Basic "+encoded);
Then use the connection as normal.
If you're using Java 7 or lower, you'll need a method to encode a String to Base64, such as:
byte[] message = (username+":"+password).getBytes("UTF-8");
String encoded = javax.xml.bind.DatatypeConverter.printBase64Binary(message);
Yes, that's all you have to do in order to use Basic Auth. The code above to set the Request Property should be done immediately after opening the connection and before getting the Input or Output streams.
Incidentally, in case someone else runs into the same, the android problem, is also present if you use org.apache.commons.codec.binary.Base64 and do Base64.encodeBase64String(). You need to do Base64.encodeBase64() and get a byte[] then construct the string.
It caught me offguard entirely that the results would be different for the line ending between those two methods.
You can use java.net.Authenticator to configure basic auth. globally for every request send by your application, see :
http://docs.oracle.com/javase/7/docs/technotes/guides/net/http-auth.html
http://developer.android.com/reference/java/net/Authenticator.html#getPasswordAuthentication()
you need to do this just copy paste it be happy
HttpURLConnection urlConnection;
String url;
// String data = json;
String result = null;
try {
String username ="danish.hussain#gmail.com";
String password = "12345678";
String auth =new String(username + ":" + password);
byte[] data1 = auth.getBytes(UTF_8);
String base64 = Base64.encodeToString(data1, Base64.NO_WRAP);
//Connect
urlConnection = (HttpURLConnection) ((new URL(urlBasePath).openConnection()));
urlConnection.setDoOutput(true);
urlConnection.setRequestProperty("Content-Type", "application/json");
urlConnection.setRequestProperty("Authorization", "Basic "+base64);
urlConnection.setRequestProperty("Accept", "application/json");
urlConnection.setRequestMethod("POST");
urlConnection.setConnectTimeout(10000);
urlConnection.connect();
JSONObject obj = new JSONObject();
obj.put("MobileNumber", "+97333746934");
obj.put("EmailAddress", "danish.hussain#dhl.com");
obj.put("FirstName", "Danish");
obj.put("LastName", "Hussain");
obj.put("Country", "BH");
obj.put("Language", "EN");
String data = obj.toString();
//Write
OutputStream outputStream = urlConnection.getOutputStream();
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(outputStream, "UTF-8"));
writer.write(data);
writer.close();
outputStream.close();
int responseCode=urlConnection.getResponseCode();
if (responseCode == HttpsURLConnection.HTTP_OK) {
//Read
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "UTF-8"));
String line = null;
StringBuilder sb = new StringBuilder();
while ((line = bufferedReader.readLine()) != null) {
sb.append(line);
}
bufferedReader.close();
result = sb.toString();
}else {
// return new String("false : "+responseCode);
new String("false : "+responseCode);
}
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (JSONException e) {
e.printStackTrace();
}
I was having this problem too.
And Now I have solved this problem.
My code is :
URL url = new URL(stringUrl);
String authStr = "MyAPIKey"+":"+"Password";
System.out.println("Original String is " + authStr);
// encode data on your side using BASE64
byte[] bytesEncoded = Base64.encodeBase64(authStr .getBytes());
String authEncoded = new String(bytesEncoded);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("Authorization", "Basic "+authEncoded);
It may help many others.
Best of luck.
Regarding the Base64 encoding problem, I found this library: http://sourceforge.net/projects/migbase64/
I have not fully vetted it but I am using it for the Basic Authentication solution shown above (as well as for image encoding/decoding), and it works well. It provides a parameter for whether or not to include the newline.