We Keep Coding

Taking text from a response web page using Java

I am sending commands to a server using http, and I currently need to parse a response that the server sends back (I am sending the command via the command line, and the servers response appears in my browser).
There are a lot of resources such as this: Saving a web page to a file in Java, that clearly illustrate how to scrape a page such as cnn.com. However, since this is a response page that is only generated when the camera receives a specific command, my attempts to use the method described by Mike Deck (in the link above) have met with failure. (Specifically, when my program requests the page again the server returns a 401 error.)
The response from the server opens a new tab in my browser. Essentially, I need to know how to save the current web page using java, since reading in a file is probably the most simple way to approach this. Do any of you know how to do this?
TL;DR How do you save the current webpage to a webpage.html or webpage.txt file using java?
EDIT: I used Base64 from the Apache commons codec, which solved my 401 authentication issue. However, I am still getting a 400 error when I attempt to connect my InputStream (see below). Does this mean a connection isn't being established in the first place?
URL url = new URL ("http://"+ipAddress+"/axis-cgi/record/record.cgi?diskid=SD_DISK");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String (encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoInput (true);
connection.setRequestProperty ("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream)connection.getInputStream();
BufferedReader in = new BufferedReader (new InputStreamReader (content));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
EDIT 2: Changing the request to a GET resolved the issue.

So while scrutinizing my code above, I decided to change
connection.setRequestMethod("POST");
to
connection.setRequestMethod("GET");
This solved my problem. In hindsight, I think the server was not recognizing the HTTP because it is not set up to handle the various trappings that come along with post.

Methods for downloading files from a URL in Java

I want to download a binary file from a webserver which is running on a embedded device. That file can be downloaded manually after a basic http authentication by using that
URL: http://10.10.10.10/config.bin... Now i want to automate this process with a simple Java application. By using the basic java tools i managed to download the file:
URL mlrrl = new URL(url);
HttpURLConnection con = (HttpURLConnection) mlrUrl.openConnection();
con.setRequestMethod("GET");
con.setAllowUserInteraction(false);
con.setDoInput(true);
con.setDoOutput(true);
con.setConnectTimeout(10000);
con.setRequestProperty("Authorization", "Basic " + authStringEnc);
InputStream stream = con.getInputStream();
BufferedInputStream in = new BufferedInputStream(stream);
FileOutputStream file = new FileOutputStream("configDown.bin");
BufferedOutputStream out = new BufferedOutputStream(file);
int i;
while ((i = in.read()) != -1) {
out.write(i);
}
out.flush();
That seemed to work fine! In comparison with the manually (by Browser-Webinterface) downloaded file the generated file have the same length, but after opening both binaries with a hex-editor they were different.
My questions:
Any ideas why the hex-codes are different? (Maybe missing connection properties?)
There is a copyUrlToFile-Method in the apache.commons package. Is it possible to use it with basic http authentification, as a alternative?
Are there other java libs or tools for downloading files from urls?

IMHO using the (deprecated) Apache Commons HttpClient ( http://hc.apache.org/httpclient-3.x/ )is the simplest way to do this. Alternatively you can use the new and (supposedly) improved HttpComponents ( http://hc.apache.org/ ).

I can´t explain why, but the faulty property "setDoOutput" = "true" maybe influenced the content of the downloaded file. Setting the value to "false" solved that issue for me!

Better way to send lots of text to a server?

I have a java application that sends text to a sql database on a server. Currently my java application takes the text, puts it into the url, then sends it to a php page on the server that takes it with GET and puts it in the database. that works fine to an extent, the problem is, that i need to be able to send lots of text, and i keep getting 414, uri to long errors. is there a better way to do this?
ok, i tried what you said, and read the tutorial, but something is not working. here is my code that i tried
public void submitText(String urls,String data) throws IOException{
URL url = new URL(urls);
URLConnection con = url.openConnection();
con.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
con.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
out.write(data);
out.flush();
}
submitText(server + "publicPB.php", "param=" + text);
here is my php code
$param = $_POST['param'];
$sql = "UPDATE table SET cell='{$param}' WHERE 1";
mysql_query($sql);
...
im pretty sure its not a problem with the php as the php worked fine with GET, and thats all i change with it, my problem i think is that im not 100% sure how to send data to it with the java

Use a POST instead of a GET and send the text as the request body. You can only pass so much data to a URL. E.g.:
// Assuming 'input' is a String and contains your text
URL url = new URL("http://hostname/path");
URLConnection con = url.openConnection();
con.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
con.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
out.write(input);
out.close();
See Reading from and Writing to a URLConnection for more details.

Why don't you use POST to send data across to PHP page? GET does have a smaller limit of content.

Use POST requests, which do not have content length limits.

POST requests do not have length content limits and are much secure than GET requests ;)

If using SQL Server I would look into leveraging BCP. You can write the file and call BCP from within Java, and it will send the information directly to your database.

Writing post data from one java servlet to another

I am trying to write a servlet that will send a XML file (xml formatted string) to another servlet via a POST.
(Non essential xml generating code replaced with "Hello there")
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();
This is causing a server error, and the second servlet is never invoked.

This kind of thing is much easier using a library like HttpClient. There's even a post XML code example:
PostMethod post = new PostMethod(url);
RequestEntity entity = new FileRequestEntity(inputFile, "text/xml; charset=ISO-8859-1");
post.setRequestEntity(entity);
HttpClient httpclient = new HttpClient();
int result = httpclient.executeMethod(post);

I recommend using Apache HTTPClient instead, because it's a nicer API.
But to solve this current problem: try calling connection.setDoOutput(true); after you open the connection.
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();

The contents of an HTTP post upload stream and the mechanics of it don't seem to be what you are expecting them to be. You cannot just write a file as the post content, because POST has very specific RFC standards on how the data included in a POST request is supposed to be sent. It is not just the formatted of the content itself, but it is also the mechanic of how it is "written" to the outputstream. Alot of the time POST is now written in chunks. If you look at the source code of Apache's HTTPClient you will see how it writes the chunks.
There are quirks with the content length as result, because the content length is increased by a small number identifying the chunk and a random small sequence of characters that delimits each chunk as it is written over the stream. Look at some of the other methods described in newer Java versions of the HTTPURLConnection.
http://java.sun.com/javase/6/docs/api/java/net/HttpURLConnection.html#setChunkedStreamingMode(int)
If you don't know what you are doing and don't want to learn it, dealing with adding a dependency like Apache HTTPClient really does end up being much easier because it abstracts all the complexity and just works.

Don't forget to use:
connection.setDoOutput( true)
if you intend on sending output.

Does new URL(...).openConnection() necessarily imply a POST?

If I create an HTTP java.net.URL and then call openConnection() on it, does it necessarily imply that an HTTP post is going to happen? I know that openStream() implies a GET. If so, how do you perform one of the other HTTP verbs without having to work with the raw socket layer?

If you retrieve the URLConnection object using openConnection() it doesn't actually start communicating with the server. That doesn't happen until you get the stream from the URLConnection(). When you first get the connection you can add/change headers and other connection properties before actually opening it.
URLConnection's life cycle is a bit odd. It doesn't send the headers to the server until you've gotten one of the streams. If you just get the input stream then I believe it does a GET, sends the headers, then lets you read the output. If you get the output stream then I believe it sends it as a POST, as it assumes you'll be writing data to it (You may need to call setDoOutput(true) for the output stream to work). As soon as you get the input stream the output stream is closed and it waits for the response from the server.
For example, this should do a POST:
URL myURL = new URL("http://example.com/my/path");
URLConnection conn = myURL.openConnection();
conn.setDoOutput(true);
conn.setDoInput(true);
OutputStream os = conn.getOutputStream();
os.write("Hi there!");
os.close();
InputStream is = conn.getInputStream();
// read stuff here
While this would do a GET:
URL myURL = new URL("http://example.com/my/path");
URLConnection conn = myURL.openConnection();
conn.setDoOutput(false);
conn.setDoInput(true);
InputStream is = conn.getInputStream();
// read stuff here
URLConnection will also do other weird things. If the server specifies a content length then URLConnection will keep the underlying input stream open until it receives that much data, even if you explicitly close it. This caused a lot of problems for us as it made shutting our client down cleanly a bit hard, as the URLConnection would keep the network connection open. This probably probably exists even if you just use getStream() though.

No it does not. But if the protocol of the URL is HTTP, you'll get a HttpURLConnection as a return object. This class has a setRequestMethod method to specify which HTTP method you want to use.
If you want to do more sophisticated stuff you're probably better off using a library like Jakarta HttpClient.