We Keep Coding

HTTPUrl connection closed automatically

i am reading a xml response using httpurlconnection. i parsed the response using JAXB. I didnt close the connection. When i again try to read from the URL , I am getting the error as Input stream. Do i have to open connection twice or is there any way to open connection once and read the response twice and then close the connection?

JAXB likely consumes the InputStream and then closes it. You would need to use some kind of FilterInputStream so that it's buffered and reusable.
With Guava, you can do something like
HttpURLConnection con = ...; // get it
InputStream in = con.getInputStream();
String content = CharStreams.toString(new InputStreamReader(in, Charsets.UTF_8));
Then pass a new InputStream to JAXB made from the bytes of content.
InputStream inForJAXB = new ByteArrayInputStream(content.getBytes());
You can do the same thing again for any other component that needs the content of the HttpURLConnection input stream.

HTTP post body lost spuriously (HttpURLConnection & Jetty)

I have a home grown protocol which uses HttpURLConnection (from Java 1.6) & Jetty (6.1.26) to POST a block of xml as a request and receive a block of xml as a response. The amounts of xml are approx. 5KB.
When running both sender and receiver on Linux EC2 instances in different parts of the world I'm finding that in about 0.04% of my requests the Jetty handler sees the xml request (the post body) as an empty string. I've checked and the client outputs that it's consistently trying to send the correct (> 0 length) xml request string.
I have also reproduced this by looping my JUnit tests on my local (Win 8) box.
I assume the error must be something like:
Misuse of buffers
An HttpURLConnection bug
A network error
A Jetty bug
A random head slapping stupid thing I've done in the code
The relevant code is below:
CLIENT
connection = (HttpURLConnection) (new URL (url)).openConnection();
connection.setReadTimeout(readTimeoutMS);
connection.setConnectTimeout(connectTimeoutMS);
connection.setRequestMethod("POST");
connection.setAllowUserInteraction(false);
connection.setDoOutput(true);
// Send request
byte[] postBytes = requestXML.getBytes("UTF-8");
connection.setRequestProperty("Content-length", "" + postBytes.length);
OutputStream os = connection.getOutputStream();
os.write(postBytes);
os.flush();
os.close();
// Read response
InputStream is = connection.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
is.close();
connection.disconnect();
return writer.toString();
SERVER (Jetty handler)
public void handle(java.lang.String target, javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response, int dispatch) {
InputStream is = request.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
is.close();
String requestXML = writer.toString();
// requestXML is 0 length string about 0.04% of time
Can anyone think of why I'd randomly get the request as an empty string?
Thanks!
EDIT
I introduced some more trace and getContentLength() returns -1 when the error occurs, but the client output still shows it's sending the right amount of bytes.

I can't think of why you are getting a empty string. Code looks correct. If you update you code to check for empty string and if found report the content-length and transfer-encoding of the request, that would be helpful to identify the culprit. A wireshark trace of the network data would also be good.
But the bad new is that jetty-6 is really end of life, and we are unlikely to be updating it. If you are writing the code today, then you really should be using jetty-7 or 8. Perhaps even jetty-9 milestone release if you are brave. If you find such and error in jetty-9, I'd be all over it like a rash trying to fix it for you!

Make sure you set connection.setRequestProperty("Content-Type", "application/xml"); It's possible POST data may be discarded without some Content-type. This was the case when I replicated your problem locally (against a Grails embedded Tomcat instance), and supplying this fixed it.

Java's URLConnection not receiving entire binary file

I am currently working on a school that encompasses creating a P2P client for a standard we came up with in class that uses HTTP to request chunks of a binary file from peers. We are allowed to us Java's HTTP libraries to make these requests, however I am hitting a major problem with these libraries. All chunks of a file will be served up in chunks that are <=64KB, but when I use the following code, the max amount of bytes that I receive is around 15040 even though the content-length of the response is 64KB:
String response = "";
URL url = new URL(uriPath);
URLConnection conn = url.openConnection ();
conn.setConnectTimeout(30 * 1000);
conn.setReadTimeout(30 * 1000);
InputStream stream = conn.getInputStream();
ByteArrayOutputStream byteArrayOut = new ByteArrayOutputStream();
int c;
byte[] buffer = new byte[4096];
while ((c = stream.read(buffer)) != -1)
{
byteArrayOut.write(buffer,0,c);
}
body = byteArrayOut.toByteArray();
stream.close();
result.put(Constants.HEADER_CONTENT_LENGTH, conn.getHeaderField(Constants.HEADER_CONTENT_LENGTH));
result.put(Constants.HEADER_CONTENT_CHECKSUM, conn.getHeaderField(Constants.HEADER_CONTENT_CHECKSUM));
result.put(Constants.KEY_BODY, new String(body));
We've tested our server component, and that serves the file correctly when accessing a chunk with wget or in a browser - this java client is the only problematic client we were able to find.
Is this a problem with Java's URLConnection class, or is there something in my code that is wrong with reading a binary file that is returned in a response?
Note: I am using Java 1.6 in Eclipse and from the command line.

How do you know that the max amount of bytes is 15040? Did you byteArrayOut.toByteArray().length or did you do new String(byteArrayOut.toByteArray()).length()?
Creating a new String from a byte array that has binary content is likely to give unpredictable results. Use a FileOutputStream and open the file.

Determine size of HTTP Response?

Is there a way to determine the size of the HTTPServletResponse content? I read this get-size-of-http-response-in-java question but sadly where I work I do not have access to CommonsIO :(
The response content consists of a single complex object so I have considered writing it out to a temp file and then checking that file. This is not something I want to be doing as a diagnostic while the application is running in production though so want to avoid it if at all possible.
PS I read erickson's answer but it mentioned input streams I want to know the size of the object being written out... Would be really nice if the writeObject() method returned a number representing bytes written instead of void...

If you have access to the response header, you can read the Content-Length.
Here is a example of a response header:
(Status-Line):HTTP/1.1 200 OK
Connection:Keep-Alive
Date:Fri, 25 Mar 2011 16:26:56 GMT
Content-Length:728
Check this out: Header Field Definitions

This seems to be what you're looking for:
DataOutputStream dos = new DataOutputStream(response.getOutputStream());
...
int len = dos.size();

I eventually found a way to get what I wanted:
URLConnection con = servletURL.openConnection();
BufferedInputStream bif = new BufferedInputStream(con.getInputStream());
ObjectInputStream input = new ObjectInputStream(bif);
int avail = bif.available();
System.out.println("Response content size = " + avail);
This allowed me to see the response size on the client. I still would like to know what it is on the server side before it is sent but this was the next best thing.

Assuming the use of ObjectOutputStream, build it around a java.io.ByteArrayOutputStream:
ByteArrayOutputStream contentBytes = new ByteArrayOutputStream();
ObjectOutputStream objectOut = new ObjectOutputStream(contentBytes);
objectOut.writeObject(content);
int contentLength = contentBytes.size();
And then you can send the content with
contentBytes.writeTo(connection.getOutputStream());
where connection is whatever you're getting your OutputStream from.
Better late than never, right?

Writing post data from one java servlet to another

I am trying to write a servlet that will send a XML file (xml formatted string) to another servlet via a POST.
(Non essential xml generating code replaced with "Hello there")
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();
This is causing a server error, and the second servlet is never invoked.

This kind of thing is much easier using a library like HttpClient. There's even a post XML code example:
PostMethod post = new PostMethod(url);
RequestEntity entity = new FileRequestEntity(inputFile, "text/xml; charset=ISO-8859-1");
post.setRequestEntity(entity);
HttpClient httpclient = new HttpClient();
int result = httpclient.executeMethod(post);

I recommend using Apache HTTPClient instead, because it's a nicer API.
But to solve this current problem: try calling connection.setDoOutput(true); after you open the connection.
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();

The contents of an HTTP post upload stream and the mechanics of it don't seem to be what you are expecting them to be. You cannot just write a file as the post content, because POST has very specific RFC standards on how the data included in a POST request is supposed to be sent. It is not just the formatted of the content itself, but it is also the mechanic of how it is "written" to the outputstream. Alot of the time POST is now written in chunks. If you look at the source code of Apache's HTTPClient you will see how it writes the chunks.
There are quirks with the content length as result, because the content length is increased by a small number identifying the chunk and a random small sequence of characters that delimits each chunk as it is written over the stream. Look at some of the other methods described in newer Java versions of the HTTPURLConnection.
http://java.sun.com/javase/6/docs/api/java/net/HttpURLConnection.html#setChunkedStreamingMode(int)
If you don't know what you are doing and don't want to learn it, dealing with adding a dependency like Apache HTTPClient really does end up being much easier because it abstracts all the complexity and just works.

Don't forget to use:
connection.setDoOutput( true)
if you intend on sending output.