We Keep Coding

Java program is not able to write complete web content, however in SOAPUI, I can see complete content

I wrote a Java program to make a simple HTTP get call and read content from a REST webservice. I include 2 custom HTTP headers and provide values to them.
In SOAPUI, when I make a REST call with updating those 2 headers, I get proper response, however when I make the same call in my Java program, I get truncated output. Below is my code:
try {
URL url = new URL(lnk);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("HeaderParameter1", param1);
connection.setRequestProperty("HeaderParamter2", param2);
connection.setConnectTimeout(60000);
InputStream is = connection.getInputStream();
byte [] b = new byte [is.available()];
is.read(b);
String input = new String(b);
System.out.println("The payload is \n"+input);
connection.disconnect();} catch(Exception e){}

Calling is.available() returns the number of bytes that are available to be read without blocking, but that may be less than the full content of the response. In the documentation for this method, it says: "It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream."
A simple way to read the entire response into a String could be:
InputStream is = connection.getInputStream();
byte [] b = is.readAllBytes();
String input = new String(b);
There are two caveats with this approach:
The new String(byte[]) constructor uses the platform's default charset to convert the bytes to characters. This is generally not a good idea for network protocols, since the client and server need to agree on the character encoding. You can hardcode the charset encoding if it will always be the same, for example: new String(b, "UTF-8").
If the response is very large, this will allocate a large byte array. If your handling of the response can be written in a streaming manner, it would be better to read iteratively with a fixed buffer size, as Anders mentioned in his comment.

Use socket to get picture from web Java

I am doing a project to get the picture from a website(anyone will be OK),and I know that I could use the URL to get it . But I want to know better about the TCP ,so I use the socket to get it . That's all be OK, but the problem is that the data stream I received contain the respond of the HTTP ,and I don't know how to filter it.
Here is my code (just a part of it)
Socket socket = new Socket(netAdress, 80);
bw = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
bw.write("GET HTTP://" + sources + " HTTP/1.0\r\n");
bw.write("\r\n");
bw.flush()//connect
BufferedOutputStream writeImg = new BufferedOutputStream(new FileOutputStream(adj));
byte[] data = new byte[512];
int len=0;
boolean OK=false;
while ((len=in.read(data))>0) {
writeImg.write(data,0,len);
writeImg.flush();
}//receive the data stream
and this is what I received,and the picture couldn't open.
the detail of the data stream
If you know how to solved the problem or you have a better idea of get the picture By socket ,please contact me.Thanks.

... this is what I received,and the picture couldn't open
Yup. The response starts with an HTTP response header.
If you know how to solved the problem ...
Well, this is a hack, and NOT recommended (and it won't work in general!) but the HTTP response header ends with the first <CR> <NL> <CR> <NL> sequence (ASCII control codes). So if you strip off everything up to and including that sequence you should have an image. (Unless it is compressed, or encoded, or a multi-part, or .....)
... or you have a better idea of get the picture.
A better idea is to use the URL. Seriously.

Yahoo YQL bad request

I'm trying to use Yahoo Content Analysis using a file containing text as input. So every character and length is possible.
This code works with a simple text String (no special characters, short text) however when I use longer texts or special characters I get a Bad Request error (HTTP 400) sometimes with an error message like "no viable alternative at character '['" or without an error message.
I encode every request and HTTP Post shouldn't have any limit as to the length.
Does the Yahoo service place a limit on the length of the request and/or are there any characters that it can't handle?
Any help to help this work is appreciated!
Here's my code (using commons-httpclient):
String fileInput = FileUtils.readFileToString(f);
StringBuilder builder = new StringBuilder();
builder.append("http://query.yahooapis.com/v1/public/yql?");
System.out.println(fileInput);
builder.append("q=")
.append(URLEncoder.encode("select * from contentanalysis.analyze where text='"+ fileInput +"'" , "UTF-8"))
.append("&format=json");
final String postUrl = builder.toString();
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(postUrl);
// Send POST request
int statusCode = client.executeMethod(method);

I think the problem is that while you are sending the request as an HTTP POST, the YQL query and text are all included in the URL. YQL does not really have a way for you to make HTTP POST requests directly, so I can think of a couple options:
Directly use the Content Analysis web service with an HTTP POST (docs)
Create a custom YQL data table which uses the <execute> tag to run custom JavaScript which could do the POST (example)
Of these options I think the former would be easier.

Multipart base64 image posting in java

I have a httpurlconnection code that is meant to post data to server. When I am trying to send small data it's ok but whene i try to send a base64 image string the string is not received properly at server. I have checked the base64 converted string from jpg and the encoding is ok. So, there is something wrong with the http process that's making the string corrupted. Initially I thought that probably it's the big images that's causing the problem but on using a very small image only transfers part of the image to server and the rest is corrupted.
Can somebody suggest me how can I send a base64 image string to server. My code is as below:
HttpURLConnection hpcon= (HttpURLConnection) Url.openConnection();
hpcon.setRequestMethod("POST");
hpcon.setConnectTimeout(5 * 1000);
hpcon.setDoOutput(true);
OutputStreamWriter osw= new OutputStreamWriter(hpcon.getOutputStream());
osw.write(data);
osw.close();
The data is something like:
key=YeP1r&nodeId=5&typeId=9&status=0&eventImage=imageString

Couldn't find a straight solution using HttpUrlConnection.
Used: httpClient from apache.
Actually pretty easy to use.
Last example multipart

Problem reading request body in servlet

I'am writing a HTTP proxy that is part of a test/verification
system. The proxy filters all requests coming from the client device
and directs them towards various systems under test.
The proxy is implemented as a servlet where each request is forwarded
to the target system, it handles both GET and POST. Somtimes the
response from the target system is altered to fit various test
conditions, but that is not the part of the problem.
When forwarding a request, all headers are copied except for those
that is part of the actual HTTP transfer such as Content-Length and
Connection headers.
If the request is a HTTP POST, then the entity body of the request is
forwarded as well and here is where it doesnt work sometimes.
The code reading the entity body from the servlet request is the following:
URL url = new URL(targetURL);
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
String method = request.getMethod();
java.util.Enumeration headers = request.getHeaderNames();
while(headers.hasMoreElements()) {
String headerName = (String)headers.nextElement();
String headerValue = request.getHeader(headerName);
if (...) { // do various adaptive stuff based on header
}
conn.setRequestProperty(headerName, headerValue);
}
// here is the part that fails
char postBody[] = new char[1024];
int len;
if(method.equals("POST")) {
logger.debug("guiProxy, handle post, read request body");
conn.setDoOutput(true);
BufferedReader br = request.getReader();
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream()));
do {
logger.debug("Read request into buffer of size: " + postBody.length);
len = br.read(postBody, 0, postBody.length);
logger.debug("guiProxy, send request body, got " + len + " bytes from request");
if(len != -1) {
bw.write(postBody, 0, len);
}
} while(len != -1);
bw.close();
}
So what happends is that the first time a POST is received, -1
characters are read from the request reader, a wireshark trace shows
that the entity body containing URL encoded post parameters are there
and it is in one TCP segment so there is no network related
differences.
The second time, br.read successfully returns the 232 bytes in the
POST request entity body and every forthcoming request works as well.
The only difference between the first and forthcoming POST requests is
that in the first one, no cookies are present, but in the second one,
a cookie is present that maps to the JSESSION.
Can it be a side effect of entity body not being available since the
request processing in the servlet container allready has read the POST
parameters, but why does it work on forthcoming requests.
I believe that the solution is of course to ignore the entity body on
POST requests containing URL encoded data and fetch all parameters
from the servlet request instead using getParameter and reinsert them
int the outgoing request.
Allthough that is tricky since the POST request could contain GET
parameters, not in our application right now, but implementing it
correctly is some work.
So my question is basically: why do the reader from
request.getReader() return -1 when reading and an entity body is
present in the request, if the entity body is not available for
reading, then getReader should throw an illegal state exception. I
have also tried with InputStream using getInputStream() with the same
results.
All of this is tested on apache-tomcat-6.0.18.

So my question is basically: why do the reader from request.getReader() return -1 when reading.
It will return -1 when there is no body or when it has already been read. You cannot read it twice. Make sure that nothing before in the request/response chain has read it.
and an entity body is present in the request, if the entity body is not available for reading, then getReader should throw an illegal state exception.
It will only throw that when you have already called getInputStream() on the request before, not when it is not available.
I have also tried with InputStream using getInputStream() with the same results.
After all, I'd prefer streaming bytes than characters because you then don't need to take character encoding into account (which you aren't doing as far now, this may lead to future problems when you will get this all to work).

Seems, that moving
BufferedReader br = request.getReader()
before all operations, that read request (like request.getHeader() ), works for me well .