We Keep Coding

Taking text from a response web page using Java

I am sending commands to a server using http, and I currently need to parse a response that the server sends back (I am sending the command via the command line, and the servers response appears in my browser).
There are a lot of resources such as this: Saving a web page to a file in Java, that clearly illustrate how to scrape a page such as cnn.com. However, since this is a response page that is only generated when the camera receives a specific command, my attempts to use the method described by Mike Deck (in the link above) have met with failure. (Specifically, when my program requests the page again the server returns a 401 error.)
The response from the server opens a new tab in my browser. Essentially, I need to know how to save the current web page using java, since reading in a file is probably the most simple way to approach this. Do any of you know how to do this?
TL;DR How do you save the current webpage to a webpage.html or webpage.txt file using java?
EDIT: I used Base64 from the Apache commons codec, which solved my 401 authentication issue. However, I am still getting a 400 error when I attempt to connect my InputStream (see below). Does this mean a connection isn't being established in the first place?
URL url = new URL ("http://"+ipAddress+"/axis-cgi/record/record.cgi?diskid=SD_DISK");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String (encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoInput (true);
connection.setRequestProperty ("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream)connection.getInputStream();
BufferedReader in = new BufferedReader (new InputStreamReader (content));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
EDIT 2: Changing the request to a GET resolved the issue.

So while scrutinizing my code above, I decided to change
connection.setRequestMethod("POST");
to
connection.setRequestMethod("GET");
This solved my problem. In hindsight, I think the server was not recognizing the HTTP because it is not set up to handle the various trappings that come along with post.

Java - Unable to write on URLConnection

I need to add contents on file existing in tomcat server. So, I am using URLConnection to do this task.
Code I am trying:
URL url = new URL("http://localhost:8080/css/extractedcss.css");
URLConnection urlcon = url.openConnection();
urlcon.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(
urlcon.getOutputStream());
out.write("New Text");
out.close();
No any exception I am getting during execution of above code but when I look into the file, no any new text I am getting.
Please help!
Regards,

You can't write directly to a file in your tomcat server - at least, not in HTTP you can't.
You'll have to write a servlet to do the writing for you, and then use a POST/PUT request to this servlet with the data you want written.

Download file programmatically

I am trying to download an vcalendar using a java application, but I can't download from a specific link.
My code is:
URL uri = new URL("http://codebits.eu/s/calendar.ics");
InputStream in = uri.openStream();
int r = in.read();
while(r != -1) {
System.out.print((char)r);
r = in.read();
}
When I try to download from another link it works (ex: http://www.mysportscal.com/Files_iCal_CSV/iCal_AUTO_2011/f1_2011.ics). Something don't allow me to download and I can't figure out why, when I try with the browser it works.

I'd follow this example. Basically, get the response code for the connection. If it's a redirect (e.g. 301 in this case), retrieve the header location and attempt to access the file using that.
Simplistic Example:
URL uri = new URL("http://codebits.eu/s/calendar.ics");
HttpURLConnection con = (HttpURLConnection)uri.openConnection();
System.out.println(con.getResponseCode());
System.out.println(con.getHeaderField("Location"));
uri = new URL(con.getHeaderField("Location"));
con = (HttpURLConnection)uri.openConnection();
InputStream in = con.getInputStream();

You should check what that link actually provides. For example, it might be a page that has moved, which gives you back an HTTP 301 code. Your browser will automatically know to go and fetch it from the new URL, but your program won't.
You might want to try, for example, wireshark to sniff the actual traffic when you do the browser request.

I think too that there is a redirect. The browser downloads from ssl secured https://codebits.eu/s/calendar.ics. Try using a HttpURLConnection, it should follow redirects automatically:
HttpURLConnection con = (HttpURLConnection)uri.openConnection();
InputStream in = con.getInputStream();

How can I download comments on a webpage (Android)

Usually I use this code to download a webpage source:
URL myURL = new URL("http://mysite.com/index.html");
StringBuffer all = new StringBuffer("");
URLConnection ucon = myURL.openConnection();
InputStream is = ucon.getInputStream();
BufferedReader page = new BufferedReader(new InputStreamReader(is, "ISO-8859-15"));
while((linea = page.readLine()) != null){
all.append(linea.trim());
}
It works fine with a wifi connection because it downloads the string like <!-- it's a comment -->,but i tried to used a mobile connection with my mobile phone but it doesn't download the comments.. Is there a method to include the comments on download webpage source?
thx for reply ;)

It is possible that your service provider is compressing the pages on their side to reduce the data sent. I've not heard of this being done for HTML but it is frequently done for JPG, so it's easy to image that's what's happening. This compression would be very likely to remove comments.
It would be nice if there was some http convention to tell the stack 'never compress', but (at fas as I know) there is not. So you're probably out of luck.

Writing post data from one java servlet to another

I am trying to write a servlet that will send a XML file (xml formatted string) to another servlet via a POST.
(Non essential xml generating code replaced with "Hello there")
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();
This is causing a server error, and the second servlet is never invoked.

This kind of thing is much easier using a library like HttpClient. There's even a post XML code example:
PostMethod post = new PostMethod(url);
RequestEntity entity = new FileRequestEntity(inputFile, "text/xml; charset=ISO-8859-1");
post.setRequestEntity(entity);
HttpClient httpclient = new HttpClient();
int result = httpclient.executeMethod(post);

I recommend using Apache HTTPClient instead, because it's a nicer API.
But to solve this current problem: try calling connection.setDoOutput(true); after you open the connection.
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();

The contents of an HTTP post upload stream and the mechanics of it don't seem to be what you are expecting them to be. You cannot just write a file as the post content, because POST has very specific RFC standards on how the data included in a POST request is supposed to be sent. It is not just the formatted of the content itself, but it is also the mechanic of how it is "written" to the outputstream. Alot of the time POST is now written in chunks. If you look at the source code of Apache's HTTPClient you will see how it writes the chunks.
There are quirks with the content length as result, because the content length is increased by a small number identifying the chunk and a random small sequence of characters that delimits each chunk as it is written over the stream. Look at some of the other methods described in newer Java versions of the HTTPURLConnection.
http://java.sun.com/javase/6/docs/api/java/net/HttpURLConnection.html#setChunkedStreamingMode(int)
If you don't know what you are doing and don't want to learn it, dealing with adding a dependency like Apache HTTPClient really does end up being much easier because it abstracts all the complexity and just works.

Don't forget to use:
connection.setDoOutput( true)
if you intend on sending output.