We Keep Coding

HTTP URL connection response

I am trying to hit the URL and get the response from my Java code.
I am using URLConnection to get this response. And writing this response in html file.
When opening this html in browser after executing the java class, I am getting only google home page and not with the results.
Whats wrong with my code, my code here,
FileWriter fWriter = null;
BufferedWriter writer = null;
URL url = new URL("https://www.google.co.in/?gfe_rd=cr&ei=aS-BVpPGDOiK8Qea4aKIAw&gws_rd=ssl#q=google+post+request+from+java");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String(encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("User-Agent", "Mozilla/5.0");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setDoInput(true);
connection.setRequestProperty("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream) connection.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String line;
try {
fWriter = new FileWriter(new File("f:\\fileName.html"));
writer = new BufferedWriter(fWriter);
while ((line = in.readLine()) != null) {
String s = line.toString();
writer.write(s);
}
writer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
Same code works couple of days back, but not now.

The reason is that this url does not return search results it self. You have to understand google's working process to understand it. Open this url in your browser and view its source. You will only see lots of javascript there.
Actually, in a short summary, google uses Ajax requests to process search queries.
To perform required task you either have to use a headless browser (the hard way) which can execute javascript/ajax OR better use google search api as directed by anand.

This method of searching is not advised is supposed to fail, you must use google search APIs for this kind of work.
Note: Google uses some redirection and uses token, so even if you will find a clever way to handle it, it is ought to fail in long run.
Edit:
This is a sample of how using Google search APIs you can get your work done in reliable way; please do refer to the source for more information.
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}

Getting the count of entries in entitySet in OData using java

The following link returns the number of entries in the Customers entity Set http://services.odata.org/Northwind/Northwind.svc/Customers/$count
How to get this number using java?
URL url = new URL("http://services.odata.org/Northwind/Northwind.svc/Customers/$count");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET")
What to code after this to get the count of entries as integers?

You need to read the data from HttpURLConnection inputstream like
BufferedReader in = new BufferedReader(new InputStreamReader(
conn.getInputStream()));
String count;
while ((count = in.readLine()) != null)
//this will print the count in count variable
System.out.println(count);
in.close();
}
Note : You have to do it after you write the request to the outputstream of the HttpURLConnection. This clearly means that you write the request data to the connection's output stream and read the response data from the connection's input stream

Java - Download zip file from url

I have a problem with downloading a zip file from an url.
It works well with firefox but with my app I have a 404.
Here is my code
URL url = new URL(reportInfo.getURI().toString());
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
// Check for errors
int responseCode = con.getResponseCode();
InputStream inputStream;
if (responseCode == HttpURLConnection.HTTP_OK) {
inputStream = con.getInputStream();
} else {
inputStream = con.getErrorStream();
}
OutputStream output = new FileOutputStream("test.zip");
// Process the response
BufferedReader reader;
String line = null;
reader = new BufferedReader(new InputStreamReader(inputStream));
while ((line = reader.readLine()) != null) {
output.write(line.getBytes());
}
output.close();
inputStream.close();
Any idea ?

In Java 7, the easiest way to save a URL to a file is:
try (InputStream stream = con.getInputStream()) {
Files.copy(stream, Paths.get("test.zip"));
}

As for why you're getting a 404 - that hard to tell. You should check the value of url, which as greedybuddha says, you should get via URI.getURL(). But it's also possible that the server is using a user agent check or something similar to determine whether or not to give you the resource. You could try with something like cURL to fetch in programmatic way but without having to write any code yourself.
However, there another problem looming. It's a zip file. That's binary data. But you're using InputStreamReader, which is designed for text content. Don't do that. You should never use a Reader for binary data. Just use the InputStream:
byte[] buffer = new byte[8 * 1024]; // Or whatever
int bytesRead;
while ((bytesRead = inputStream.read(buffer)) > 0) {
output.write(buffer, 0, bytesRead);
}
Note that you should close the streams in finally blocks, or use the try-with-resources statement if you're using Java 7.

Unable to upload a Excel file using HttpURLConnection

I am successfully using the HttpURLConnection to upload *.txt files. Now I have the task to change the program for uploading every filetype. I tried my program with .xls files. The files are placed on the server, but the content isn't readable.
Like you see here:
ÃÃࡱá>þÿ
Œ¸äÌÌÌS
Sheet1Sheet2Sheet3Worksheets4 $€,þÿ3à…ŸòùOh«‘+'³Ù0Å“8Å’#x„äMicrosoft CorporationMicrosoft Excel#â€Ã¸(º»þÿ3à…ŸòùOh«‘+'³Ù0Ëœ8Å’#x„äMicrosoft CorporationMicrosoft Excel#â€Ã¸(º»#â€Ã¸(º»Ì�á°Ãâ\pMicrosoft Corporation
Here my code snippet:
HttpURLConnection urlConn = (HttpURLConnection) new URL(testdocumentURL).openConnection();
urlConn.setRequestMethod("POST");
urlConn.setRequestProperty("X-Method-Override", "PUT");
urlConn.setRequestProperty("Content-Type", "text/xml");
urlConn.setRequestProperty("Authorization", "Basic "+ Client.getPassword());
urlConn.setUseCaches(false);
urlConn.setDoInput(true);
urlConn.setDoOutput(true);
urlConn.setFollowRedirects(false);
urlConn.setRequestProperty("Slug", "Connectiontest/test.xls");
String write = readFile(test.xls);
urlConn.setRequestProperty("Content-Length","" + write.length());
urlConn.getOutputStream().write(write.getBytes("UTF8"));
In my optinion there are 2 ways to solve this problem.
change the output at .write(write.getBytes("UTF8"))
change the intput
At the moment I read the files like this:
readFile(String test){
BufferedReader reader = new BufferedReader(new FileReader(test));
String line = null;
StringBuilder stringBuilder = new StringBuilder();
String ls = System.getProperty("line.separator");
while ((line = reader.readLine()) != null) {
stringBuilder.append(line);
stringBuilder.append(ls);
}
reader.close();
return stringBuilder.toString();
}
I tried to change the RequestProperty() in this ways:
connection.setRequestProperty("content-type", "multipart/form-data");
or
connection.setRequestProperty("content-type", "application/vnd.ms-excel");
Do somebody know to Upload a .xls file like this? I have to prefere the HttpURLConnector, my boss said. :/
Thanks a lot!

Don't treat the Excel file as a String. Change readFile to read the file into a byte[] instead, then write that to the connection output stream.

Set the content type as application/vnd.ms-excel and then give it a try

How can I send POST data through url.openStream()?

i'm looking for tutorial or quick example, how i can send POST data throw openStream.
My code is:
URL url = new URL("http://localhost:8080/test");
InputStream response = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response, "UTF-8"));
Could you help me ?

URL url = new URL(urlSpec);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(method);
connection.setDoOutput(true);
connection.setDoInput(true);
// important: get output stream before input stream
OutputStream out = connection.getOutputStream();
out.write(content);
out.close();
// now you can get input stream and read.
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
writer.println(line);
}

Use Apache HTTP Compoennts http://hc.apache.org/httpcomponents-client-ga/
tutorial: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
Look for HttpPost - there are some examples of sending dynamic data, text, files and form data.

Apache HTTP Components in particular, the Client would be the best way to go.
It absracts a lot of that nasty coding you would normally have to do by hand