We Keep Coding

Java program is not able to write complete web content, however in SOAPUI, I can see complete content

I wrote a Java program to make a simple HTTP get call and read content from a REST webservice. I include 2 custom HTTP headers and provide values to them.
In SOAPUI, when I make a REST call with updating those 2 headers, I get proper response, however when I make the same call in my Java program, I get truncated output. Below is my code:
try {
URL url = new URL(lnk);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("HeaderParameter1", param1);
connection.setRequestProperty("HeaderParamter2", param2);
connection.setConnectTimeout(60000);
InputStream is = connection.getInputStream();
byte [] b = new byte [is.available()];
is.read(b);
String input = new String(b);
System.out.println("The payload is \n"+input);
connection.disconnect();} catch(Exception e){}

Calling is.available() returns the number of bytes that are available to be read without blocking, but that may be less than the full content of the response. In the documentation for this method, it says: "It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream."
A simple way to read the entire response into a String could be:
InputStream is = connection.getInputStream();
byte [] b = is.readAllBytes();
String input = new String(b);
There are two caveats with this approach:
The new String(byte[]) constructor uses the platform's default charset to convert the bytes to characters. This is generally not a good idea for network protocols, since the client and server need to agree on the character encoding. You can hardcode the charset encoding if it will always be the same, for example: new String(b, "UTF-8").
If the response is very large, this will allocate a large byte array. If your handling of the response can be written in a streaming manner, it would be better to read iteratively with a fixed buffer size, as Anders mentioned in his comment.

render video stream as partial content rather than full stream to chrome

We currently submit a video playback request to back end server the sends the full stream as inputstream to be played back in the browser(window). This works fine but has an added complication in that the seek function does not work in chrome. The suggested solution is to tell the webserver that it needs to accept a byte range and then deliver the stream in partial byte ranges. I am not sure if this will resolve the situation but my question is how to return the stream in byte ranges considering the following is the way it is done now:
InputStream is= null;
is = new FileInputStream(ndirectoryFile);
....
//(calling class request)
stream = videoWrapper.getVideo(id, address);
If I read the file in byte ranges, do I just loop through the file but how do I send the response:
InputStream is = new ByteArrayInputStream(new byte[] { 0, 1, 2 });
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[1024];
while ((nRead = is.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
byte[] byteArray = buffer.toByteArray(
The initial inputstream gets passed to quite a few classes along the way prior to sending the final response. Any ideas please.
EDIT:
I would like to just understand the html5 video issue with chrome. There are quite a few posts on setting the server response headers to include Accept-Ranges=, Content-length= , Content-Range= and this would tell chrome to download the byte range which will then allow seek feature to work. As the video playback seek works in firefox I should not have to change how I deliver the stream or would I? Would I still have to submit partial ranges of the video from the server? and how?

I resolved my situation by not having to change any streaming code but rather just ensure the correct range headers were applied to the response to ensure chrome handled the playback.
Accept-Ranges: bytes
Content-Range: bytes 0-1025/905357
Content-Length: 905357

Call large URL in Java

How should I call a large URL in Java? I'm integrating scene7 image server with java application. Here I call a URL of around 10000 characters which should return me an Image. What ways can I do this?
The way I wrote is -
URL oURL= new URL("<*LONG LONG URL - approx. 10k characters*>");
HttpURLConnection connection = (HttpURLConnection) oURL.openConnection();
InputStream stream = connection.getInputStream();
int len;
byte[] buf = new byte[1024];
BufferedImage bi = ImageIO.read(stream);
ImageIO.write(bi,"png",response.getOutputStream());
while ((len = stream.read(buf)) > 0) {
outs.write(buf, 0, len);
}

You won't get URLs that long to work reliably. The longest you can rely on working is around 2000 characters. Beyond that, you are liable to run into browser or server-side limits. (Or even limits in intermediate proxy servers)
You are going to need to "think outside the box". For example:
pass some the information that is currently encoded in your URL in the form of POST data,
pass some the information in the form of custom request headers (ewww!!), or
come up with a clever way to encode (or compress) the informational part of the URL.
All of these are likely to require changes to the server side.
Reference:
What is the maximum length of a URL in different browsers?

While the HTTP 1.1 specification does not place a limit on the length (size) of a URL, the server that you are working with might. You should check the documentation for the server you are connecting to (in this case, Adobe's Scene7.)

Just getting an image from the URL is as easy as:
URL url = new URL("<extremely long URL>");
Image image = ImageIO.read(url);
So I'm wondering if you weren't asking for something else.

Parsing XML response of SOAP call in Weblogic

We are in the process of migrating functionality from a Weblogic Server 8,1 sp5 with java 1.4 to 10.3.6 with java 1.7.
The case that is described below is working properly in the old server, however we are facing an issue when transferring the handling to the new one.
The problem lies while retrieving and parsing an XML response that was retrieved from an external system through SOAP calls.
The following libraries and procedure are used in the method:
java.net.HttpURLConnection to make the connection
java.io.OutputStream to send the request
java.io.InputStream to get the response
byte[] to store the result before transforming in to String
javax.xml.parsers.DocumentBuilder, java.io.StringReader and org.xml.sax.InputSource to transform the String into org.w3c.dom.Document
The following exception is thrown:
"org.xml.sax.SAXParseException - Content is not allowed in trailing section."
When opening the logs of the application with notepad++ many null characters appear after the end of the file which seem to cause the issue. I repeat that no such cases appear when executing the request from the old server.
The respective code is the following:
//Creating the connection
URL u = new URL(default_server);
URLConnection uc = u.openConnection();
HttpURLConnection connection = (HttpURLConnection) uc;
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setRequestMethod(requestMethod);
connection.setRequestProperty("SOAPAction", soap_action);
OutputStream out = connection.getOutputStream();
Writer wout = new OutputStreamWriter(out);
wout.write(xmlString);
wout.flush();
wout.close();
InputStream in = connection.getInputStream();
int c = in.available();
byte r[] = new byte[c];
in.read(r);
String response = new String(r);
connection.disconnect();
//Transorming the String to XML Doc
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
StringReader theReader = new StringReader(response);
InputSource theInputSource = new InputSource();
theInputSource.setCharacterStream(theReader);
Document doc = builder.parse(theInputSource);
//Here occurs org.xml.sax.SAXParseException-Content is not allowed in trailing section
return doc;
I know that i can solve the issue by getting the stripping the response from junk characters but this not a safe resolution.
Do you have any information to share on the matter? Do you think it is a java version issue or maybe a server configuration issue?
Thank you in advance your time.
Best Regards,
George

I see two issues
in.available() as per the javadoc: Returns an estimate of the number of bytes... Do not rely on this. Loop over buffers of 8K to read the stream until you reach the end or even better, do not re-invent the wheel, use commons-io from Apache and use a single call to ÌOUtils.read
String response = new String(r); doing this you are assuming that the bytes received are encoded using the same charset as your platform encoding/charset. It is unlikely to be the case if you are on Windows or OSX. You must pass the charset and use the constructor String(byte[] bytes, Charset charset).

Determine size of HTTP Response?

Is there a way to determine the size of the HTTPServletResponse content? I read this get-size-of-http-response-in-java question but sadly where I work I do not have access to CommonsIO :(
The response content consists of a single complex object so I have considered writing it out to a temp file and then checking that file. This is not something I want to be doing as a diagnostic while the application is running in production though so want to avoid it if at all possible.
PS I read erickson's answer but it mentioned input streams I want to know the size of the object being written out... Would be really nice if the writeObject() method returned a number representing bytes written instead of void...

If you have access to the response header, you can read the Content-Length.
Here is a example of a response header:
(Status-Line):HTTP/1.1 200 OK
Connection:Keep-Alive
Date:Fri, 25 Mar 2011 16:26:56 GMT
Content-Length:728
Check this out: Header Field Definitions

This seems to be what you're looking for:
DataOutputStream dos = new DataOutputStream(response.getOutputStream());
...
int len = dos.size();

I eventually found a way to get what I wanted:
URLConnection con = servletURL.openConnection();
BufferedInputStream bif = new BufferedInputStream(con.getInputStream());
ObjectInputStream input = new ObjectInputStream(bif);
int avail = bif.available();
System.out.println("Response content size = " + avail);
This allowed me to see the response size on the client. I still would like to know what it is on the server side before it is sent but this was the next best thing.

Assuming the use of ObjectOutputStream, build it around a java.io.ByteArrayOutputStream:
ByteArrayOutputStream contentBytes = new ByteArrayOutputStream();
ObjectOutputStream objectOut = new ObjectOutputStream(contentBytes);
objectOut.writeObject(content);
int contentLength = contentBytes.size();
And then you can send the content with
contentBytes.writeTo(connection.getOutputStream());
where connection is whatever you're getting your OutputStream from.
Better late than never, right?