I'm using the following code for uploading an multiform/data form with a file upload:
URL url = new URL(DEST_URL);
String boundary = "-----------------------------" + Long.toString(System.currentTimeMillis());
PrintWriter writer = null;
URLConnection con = url.openConnection();
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestProperty("Accept-Charset", "utf-8");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/json,application/xml;q=0.9,*/*;q=0.8");
con.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
OutputStream output = con.getOutputStream();
InputStream input = new FileInputStream(new File(FILE_PATH));
writer = new PrintWriter(new OutputStreamWriter(output));
writer.println(boundary);
writer.println("Content-Disposition: form-data; name=\"input1\"");
writer.println();
writer.println("1234");
writer.flush();
writer.println(boundary);
writer.println("Content-Disposition: form-data; name=\"input1\"");
writer.println();
writer.println("asdf");
writer.flush();
writer.println(boundary);
writer.println("Content-Disposition: form-data; name=\"file1\"; filename=\"clicknpoint.png\"");
writer.println("Content-Type: image/png");
writer.println();
writer.flush();
int length = 0;
byte[] buffer = new byte[1024];
for(length = 0; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
writer.flush();
input.close();
writer.println();
writer.println(boundary + "--");
writer.flush();
input = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String cur = null;
StringBuffer buf = new StringBuffer();
while((cur = reader.readLine()) != null) {
buf.append(cur);
}
The server doesn't recognizes the params in the request. I checked with wireshark, they are there but the IP header checksum is 0x0000. I think that's problem.
Any idea where this is coming from?
The only thing I can see that might be incorrect is your use of println to generate HTTP content. Depending on your platform, the println's may be outputting a single LF character or may be outputting a CR LF pair.
HTTP definitely requires CRLF after every header. See http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4. Right now your code is at risk of violating the HTTP protocol. It might work on some platforms but not on others. Also some servers might tolerate the violation of the protocol, but others might not. The least invasive change might be changing every println to print and explicitly adding the CRLF.
writer.print("Content-Disposition: form-data; name=\"input1\"\r\n");
This may have nothing to do with your problem, but until you tighten up your conformance with HTTP your code will be at risk of failing.
As you have already realized, you are looking at two independent issues: Wireshark is showing an invalid checksum, and your server is not correctly receiving data.
Issue 1: Wireshark is showing an invalid checksum
I realize you've already figured out that this is because of TCP checksum offloading. For posterity, I'll repeat what you already know.
On modern computers, the OS does not calculate a checksum for each TCP packet it sends out. Instead, the checksum is calculated in hardware on the Network adapter. Since Wireshark is intercepting packets before they reach the network adapter, it sees the wrong checksum. To confirm that offloading is in fact the reason you are seeing invalid checksums in Wireshark, you can temporarily turn it off. For Mac OS X, use these commands in Terminal:
sudo sysctl -w net.link.ether.inet.apple_hwcksum_tx=0
sudo sysctl -w net.link.ether.inet.apple_hwcksum_rx=0
To re-enable after testing, just replace those 0s with 1s. To disable offloading on other operating systems, here in the section Step 3. If possible, disable TCP and UDP checksum offloading and TCP segmentation offloading.
Issue 2: Server is not correctly receiving data
URLConnection objects are not reusable. And calling flush() or write() on the output stream of a URLConnection immediately sends your request. After that, you must create a new URLConnection. Make sure that all your data has been added to the output stream before calling either of these methods. A really excellent discussion of the consequences of manipulating the I/O streams of URLConnections can be found here:
http://www.javaworld.com/javaworld/jw-03-2001/jw-0323-traps.html?page=4
Related
I am doing a project to get the picture from a website(anyone will be OK),and I know that I could use the URL to get it . But I want to know better about the TCP ,so I use the socket to get it . That's all be OK, but the problem is that the data stream I received contain the respond of the HTTP ,and I don't know how to filter it.
Here is my code (just a part of it)
Socket socket = new Socket(netAdress, 80);
bw = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
bw.write("GET HTTP://" + sources + " HTTP/1.0\r\n");
bw.write("\r\n");
bw.flush()//connect
BufferedOutputStream writeImg = new BufferedOutputStream(new FileOutputStream(adj));
byte[] data = new byte[512];
int len=0;
boolean OK=false;
while ((len=in.read(data))>0) {
writeImg.write(data,0,len);
writeImg.flush();
}//receive the data stream
and this is what I received,and the picture couldn't open.
the detail of the data stream
If you know how to solved the problem or you have a better idea of get the picture By socket ,please contact me.Thanks.
... this is what I received,and the picture couldn't open
Yup. The response starts with an HTTP response header.
If you know how to solved the problem ...
Well, this is a hack, and NOT recommended (and it won't work in general!) but the HTTP response header ends with the first <CR> <NL> <CR> <NL> sequence (ASCII control codes). So if you strip off everything up to and including that sequence you should have an image. (Unless it is compressed, or encoded, or a multi-part, or .....)
... or you have a better idea of get the picture.
A better idea is to use the URL. Seriously.
I am hosting a webpage from home. I made my own HTTP server using Java. This is an SSCCE:
if(command.startsWith("GET"))
{
//client is a socket on which I reply.
PrintWriter pw = new PrintWriter(client.getOutputStream(), true);
String commule = command.split(" ");
if(commule[0].equals("GET"))
{
if(commule[1].contains("."))
{
File file = new File(GEQO_SERVER_ROOT + commule[1].substring(1).replaceAll("%20", " "));
if(file.exists())
{
OutputStream out = client.getOutputStream();
InputStream stream = new FileInputStream(file);
String response = new String();
response += "HTTP/1.1 200 OK\r\n";
response += "Date: Thu, 08 Aug 2013 08:49:37 GMT\r\n";
response += "Content-Type: text/html\r\n";
response += "Content-Length: " + file.length() + "\r\n";
response += "Connection: keep-alive\r\n";
response += "\r\n";
pw.write(response); //Assume I already initialized pw as a PrintWriter
pw.flush();
copy(stream, out);
stream.close();
out.close();
}
else
{
pw.write("<html><h1>The request 404ed.</h1>");
pw.write("<body>The requested URL <b>" + commule[1] + "</b> could not be found on this server.</body></html>");
pw.flush();
}
}
else
{
BufferedReader br = new BufferedReader(new FileReader(GEQO_SERVER_ROOT + commule[1].substring(1) + "main.html"));
String sCurrentLine;
while ((sCurrentLine = br.readLine()) != null)
{
pw.print(sCurrentLine);
}
br.close();
}
}
else
{
pw.println("Unrecognized HTTP command.");
}
}
This is the main.html source :
<html>
<title>Geqo Server</title>
<body>Geqo server online and functioning!</body>
</html>
The issue is that when I try to access this page using Chrome, it displays correctly (At least when using 127.0.0.1). But when I tried accessing it on Firefox on 127.0.0.1, it works, but just gives me the html source. IE also only gives me the source. Can anyone tell me why Firefox and IE only show the source, instead of parsing it?
I think this contains some clues (Firebug screenshot) :
My source seems to be coming in a <pre> tag. I donno why, but isn't that sort of the problem?
I port-forwarded. Here's the page guys : http://110.172.170.83:17416/ (Sorry, Stackoverflow doesn't allows numerical links.)
EDIT : I found the problem. But before I explain, thanks to Bart for the SSCCE, which I used to compare with my code. This is the problem : The if statement on the eighth line if(commule[1].contains(".")) causes the code to skip the most of the code here. In that respective else block, there is even no command to send the headers. Thanks to artbristol for pointing that out.
Thanks in advance.
Your printwriter isn't flushing (as Ernest pointed out), so no HTTP headers are being sent. Look at the result of connecting directly - it just returns the raw data, with no headers.
nc 110.172.170.83 17416
GET /
<html><title>Geqo Server</title><body>Geqo server online and functioning!</body></html>
Writing an HTTP server is hard work. Unless this is for an exercise, you should use a lightweight existing one, such as Jetty, or the built in Sun HTTP server in the JDK.
Edit - A PrintWriter really isn't appropriate for doing HTTP. It's designed to deal with line-by-line data such as a file being written to disk. It's also dependent on platform-specific settings for text encoding and line endings. Check the HTTP spec for more details on how a proper HTTP server ought to work.
There would appear to be some potential issues with buffering. You write some of your output to a PrintWriter wrapper around out, and other output directly to out. I would definitely add a call to pw.flush() after the pw.write() call.
You enabled autoFlush with the second argument to
new PrintWriter(client.getOutputStream(), true)
http://docs.oracle.com/javase/7/docs/api/java/io/PrintWriter.html
Unlike the PrintStream class, if automatic flushing is enabled it will be done only when one of the println, printf, or format methods is invoked, rather than whenever a newline character happens to be output. These methods use the platform's own notion of line separator rather than the newline character.
So basically your pw.write() did not flush to the output stream. So all you need to do is replace
pw.write(response);
with
pw.println(response);
You do not send any response header.
I can't find the definition of pw in your source code?
I have a home grown protocol which uses HttpURLConnection (from Java 1.6) & Jetty (6.1.26) to POST a block of xml as a request and receive a block of xml as a response. The amounts of xml are approx. 5KB.
When running both sender and receiver on Linux EC2 instances in different parts of the world I'm finding that in about 0.04% of my requests the Jetty handler sees the xml request (the post body) as an empty string. I've checked and the client outputs that it's consistently trying to send the correct (> 0 length) xml request string.
I have also reproduced this by looping my JUnit tests on my local (Win 8) box.
I assume the error must be something like:
Misuse of buffers
An HttpURLConnection bug
A network error
A Jetty bug
A random head slapping stupid thing I've done in the code
The relevant code is below:
CLIENT
connection = (HttpURLConnection) (new URL (url)).openConnection();
connection.setReadTimeout(readTimeoutMS);
connection.setConnectTimeout(connectTimeoutMS);
connection.setRequestMethod("POST");
connection.setAllowUserInteraction(false);
connection.setDoOutput(true);
// Send request
byte[] postBytes = requestXML.getBytes("UTF-8");
connection.setRequestProperty("Content-length", "" + postBytes.length);
OutputStream os = connection.getOutputStream();
os.write(postBytes);
os.flush();
os.close();
// Read response
InputStream is = connection.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
is.close();
connection.disconnect();
return writer.toString();
SERVER (Jetty handler)
public void handle(java.lang.String target, javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response, int dispatch) {
InputStream is = request.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
is.close();
String requestXML = writer.toString();
// requestXML is 0 length string about 0.04% of time
Can anyone think of why I'd randomly get the request as an empty string?
Thanks!
EDIT
I introduced some more trace and getContentLength() returns -1 when the error occurs, but the client output still shows it's sending the right amount of bytes.
I can't think of why you are getting a empty string. Code looks correct. If you update you code to check for empty string and if found report the content-length and transfer-encoding of the request, that would be helpful to identify the culprit. A wireshark trace of the network data would also be good.
But the bad new is that jetty-6 is really end of life, and we are unlikely to be updating it. If you are writing the code today, then you really should be using jetty-7 or 8. Perhaps even jetty-9 milestone release if you are brave. If you find such and error in jetty-9, I'd be all over it like a rash trying to fix it for you!
Make sure you set connection.setRequestProperty("Content-Type", "application/xml"); It's possible POST data may be discarded without some Content-type. This was the case when I replicated your problem locally (against a Grails embedded Tomcat instance), and supplying this fixed it.
I am currently working on a school that encompasses creating a P2P client for a standard we came up with in class that uses HTTP to request chunks of a binary file from peers. We are allowed to us Java's HTTP libraries to make these requests, however I am hitting a major problem with these libraries. All chunks of a file will be served up in chunks that are <=64KB, but when I use the following code, the max amount of bytes that I receive is around 15040 even though the content-length of the response is 64KB:
String response = "";
URL url = new URL(uriPath);
URLConnection conn = url.openConnection ();
conn.setConnectTimeout(30 * 1000);
conn.setReadTimeout(30 * 1000);
InputStream stream = conn.getInputStream();
ByteArrayOutputStream byteArrayOut = new ByteArrayOutputStream();
int c;
byte[] buffer = new byte[4096];
while ((c = stream.read(buffer)) != -1)
{
byteArrayOut.write(buffer,0,c);
}
body = byteArrayOut.toByteArray();
stream.close();
result.put(Constants.HEADER_CONTENT_LENGTH, conn.getHeaderField(Constants.HEADER_CONTENT_LENGTH));
result.put(Constants.HEADER_CONTENT_CHECKSUM, conn.getHeaderField(Constants.HEADER_CONTENT_CHECKSUM));
result.put(Constants.KEY_BODY, new String(body));
We've tested our server component, and that serves the file correctly when accessing a chunk with wget or in a browser - this java client is the only problematic client we were able to find.
Is this a problem with Java's URLConnection class, or is there something in my code that is wrong with reading a binary file that is returned in a response?
Note: I am using Java 1.6 in Eclipse and from the command line.
How do you know that the max amount of bytes is 15040? Did you byteArrayOut.toByteArray().length or did you do new String(byteArrayOut.toByteArray()).length()?
Creating a new String from a byte array that has binary content is likely to give unpredictable results. Use a FileOutputStream and open the file.
Is there a way to determine the size of the HTTPServletResponse content? I read this get-size-of-http-response-in-java question but sadly where I work I do not have access to CommonsIO :(
The response content consists of a single complex object so I have considered writing it out to a temp file and then checking that file. This is not something I want to be doing as a diagnostic while the application is running in production though so want to avoid it if at all possible.
PS I read erickson's answer but it mentioned input streams I want to know the size of the object being written out... Would be really nice if the writeObject() method returned a number representing bytes written instead of void...
If you have access to the response header, you can read the Content-Length.
Here is a example of a response header:
(Status-Line):HTTP/1.1 200 OK
Connection:Keep-Alive
Date:Fri, 25 Mar 2011 16:26:56 GMT
Content-Length:728
Check this out: Header Field Definitions
This seems to be what you're looking for:
DataOutputStream dos = new DataOutputStream(response.getOutputStream());
...
int len = dos.size();
I eventually found a way to get what I wanted:
URLConnection con = servletURL.openConnection();
BufferedInputStream bif = new BufferedInputStream(con.getInputStream());
ObjectInputStream input = new ObjectInputStream(bif);
int avail = bif.available();
System.out.println("Response content size = " + avail);
This allowed me to see the response size on the client. I still would like to know what it is on the server side before it is sent but this was the next best thing.
Assuming the use of ObjectOutputStream, build it around a java.io.ByteArrayOutputStream:
ByteArrayOutputStream contentBytes = new ByteArrayOutputStream();
ObjectOutputStream objectOut = new ObjectOutputStream(contentBytes);
objectOut.writeObject(content);
int contentLength = contentBytes.size();
And then you can send the content with
contentBytes.writeTo(connection.getOutputStream());
where connection is whatever you're getting your OutputStream from.
Better late than never, right?