We Keep Coding

HTTP Server - Serving up favicon.ico

I'm playing around setting up my own java http server to better understand http servers and what goes on under the hood of the web. I've developed a pretty simple server and have been able to serve both html pages as well as data in JSON form. Then I saw the browser (I'm using chrome but assuming it's the same for others) was sending a request for favicon.ico. I'm able to identify that request on my server, so I'm trying to serve up a random icon I downloaded and resized to 16x16 pixels in png format, as that's what the internet says the size needs to be. Here's my code, note it's not supposed to be anything professional, just something that will work for my basic educational purposes:
[set up ServerSocket and listen]
public static String err_header = "HTTP/1.1 500 ERR\nAccess-Control-Allow-Origin: *";
public static String success_header = "HTTP/1.1 200 OK\nAccess-Control-Allow-Origin: *";
public static String end_header = "\r\n\r\n";
while(true){
try{
System.out.println("Listening for new connections");
clientSocket = server.accept();
System.out.println("Connection established");
InputStreamReader isr = new InputStreamReader(clientSocket.getInputStream());
BufferedReader reader = new BufferedReader(isr);
String getLine = reader.readLine();//first line of HTTP request
handleRequest(getLine,clientSocket);
}//end of try
catch(Exception e){
[error stuff]
}//end of catch
}//end of while
HandleRequest method:
public static void handleRequest(String getLine,Socket clientSocket) throws Exception{
if(getLine.substring(5,16).equals("favicon.ico")){
List<String> iconTag = new ArrayList<String>();
iconTag.add("\nContent-Type: image/png");
handleFileRequest("[file]",iconTag,clientSocket);
}//end of if
else{
handleFileRequest("[file]",clientSocket);
}//end of else
}//end of handleRequest
handleFileRequest for images:
public static void handleFileRequest(String fileName,List<String> headerTags,Socket clientSocket) throws Exception{
OutputStream out = clientSocket.getOutputStream();
BufferedReader read = new BufferedReader(new FileReader(fileName));
out.write(success_header.getBytes("UTF-8"));
Iterator<String> itr = headerTags.iterator();
while(itr.hasNext()){
out.write(itr.next().getBytes("UTF-8"));
}//end of while
out.write(end_header.getBytes("UTF-8"));
String readLine = "";
while((readLine = read.readLine())!=null){
out.write(readLine.getBytes("UTF-8"));
}//end of while
out.flush();
out.close();
}//end of handleFileRequest
And it appears to work, as the server sends the file, the browser shows the 200 OK response, but there's no favicon and when I filter network requests to just images, there is one image requested by the page being served but the favicon request is not listed there (the favicon request is in the "other" section). Similarly when clicking on the other image the image shows up on the preview, whereas that's not the case with the favicon request. Screenshot:
Meanwhile here's what the other image looks like, and it shows up in the page just fine:
I also tried including the Content-Length header, but that didn't seem to make a difference. Am I missing something obvious?
Also just to clarify, I know I can include the favicon in the actual html page, the goal isn't to do it, but to understand how it works.

Reading binary files
It seems the content of the favicon is not served correctly.
I suspect this is most likely due to the way you read its content:
while((readLine = read.readLine())!=null){
out.write(readLine.getBytes("UTF-8"));
}
Reading binary content line by line is inappropriate,
because the concept of lines, and also UTF-8 encoding,
don't make sense in the context of binary files.
And you cannot read binary content correctly line by line this way,
because the readLine method of a BufferedReader doesn't return the full line, because it strips the newline from the end.
You cannot manually add a newline character because you cannot know what exactly it was.
Here's a simpler and correct way to read the content of a binary file:
byte[] bytes = Files.readAllBytes(Paths.get("/path/to/file"));
Once you have this, it's easy to produce a correct file header with the content length, using the value of bytes.length.
What happens when you visit a page in a browser
It seems it will be good for you if we clarify a few things.
When you open a URL in a browser,
the browser sends a GET request to the web server to download the content of the original URL that you have specified.
Once it has the page content, it will send further GET requests:
Fetch a favicon if it doesn't have one already. The location of this may be specified in the HTML document, or else the browser will try to fetch SERVERNAME/favicon.ico by default
Fetch the images specified in src attribute of any (valid) <img/> tags in the document
Fetch the style sheets specified in href attribute of any (valid) <style/> tags in the document
... and similarly for <script/> tags, and so on...
The favicon is purely cosmetic, to show in browser tab titles,
the other resources are essential for rendering a page.
They are not essential in text-based browsers like lynx,
such browsers will obviously not fetch these resources.
This is the explanation for why the favicon is requested, and how.
How does a web server serve files?
In the most basic case, serving a file has two important components:
Produce an appropriate HTTP header: each line in the header is in name: value format, and each line must end with \n.
There must be at least a Content-type header.
The header must be terminated by a blank line.
After the blank line that terminates the header,
the content can be anything, even binary.
To illustrate with an example,
consider the curl command, which dumps the content of a url to standard output.
If you run curl url-to-some-html-file,
you will see the content of the html file.
If you run curl url-to-some-image-file,
you will see the content of the image file.
It will be unreadable, and your terminal will probably make funny noises.
You can redirect the output to a file with curl url-to-some-image-file > image.png,
and that will give you an image file,
binary content,
that you can open in any image viewer tool.
In short, serving files is really just printing a header on stdout,
then printing a blank line to terminate the header,
then printing the content on stdout.
Debugging the serving of an image
An easy way to debug that an image is correctly served is to save the URL to a file using curl,
and then verify that the saved file and the original file are identical,
for example using the cmp command:
curl -o file url-to-favicon
cmp file /path/to/original
The output of cmp should be empty.
This command only produces output if it finds a difference in the two files.
Implementing a simple HTTP server
Instead of using a ServerSocket,
here's a drastically simpler way to implement an HTTP server:
HttpServer server = HttpServer.create(new InetSocketAddress(1234), 0);
server.createContext("/favicon.ico", t -> {
byte[] bytes = Files.readAllBytes(Paths.get("/path/to/favicon"));
t.sendResponseHeaders(200, bytes.length);
try (OutputStream os = t.getResponseBody()) {
os.write(bytes);
}
});
server.createContext("/", t -> {
Charset charset = StandardCharsets.UTF_8;
List<String> lines = Files.readAllLines(Paths.get("/path/to/index"), charset);
t.sendResponseHeaders(200, 0);
try (OutputStream os = t.getResponseBody()) {
for (String line : lines) {
os.write((line + "\n").getBytes(charset));
}
}
});
server.start();

Use socket to get picture from web Java

I am doing a project to get the picture from a website(anyone will be OK),and I know that I could use the URL to get it . But I want to know better about the TCP ,so I use the socket to get it . That's all be OK, but the problem is that the data stream I received contain the respond of the HTTP ,and I don't know how to filter it.
Here is my code (just a part of it)
Socket socket = new Socket(netAdress, 80);
bw = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
bw.write("GET HTTP://" + sources + " HTTP/1.0\r\n");
bw.write("\r\n");
bw.flush()//connect
BufferedOutputStream writeImg = new BufferedOutputStream(new FileOutputStream(adj));
byte[] data = new byte[512];
int len=0;
boolean OK=false;
while ((len=in.read(data))>0) {
writeImg.write(data,0,len);
writeImg.flush();
}//receive the data stream
and this is what I received,and the picture couldn't open.
the detail of the data stream
If you know how to solved the problem or you have a better idea of get the picture By socket ,please contact me.Thanks.

... this is what I received,and the picture couldn't open
Yup. The response starts with an HTTP response header.
If you know how to solved the problem ...
Well, this is a hack, and NOT recommended (and it won't work in general!) but the HTTP response header ends with the first <CR> <NL> <CR> <NL> sequence (ASCII control codes). So if you strip off everything up to and including that sequence you should have an image. (Unless it is compressed, or encoded, or a multi-part, or .....)
... or you have a better idea of get the picture.
A better idea is to use the URL. Seriously.

URLConnection generates wrong IP checksum

I'm using the following code for uploading an multiform/data form with a file upload:
URL url = new URL(DEST_URL);
String boundary = "-----------------------------" + Long.toString(System.currentTimeMillis());
PrintWriter writer = null;
URLConnection con = url.openConnection();
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestProperty("Accept-Charset", "utf-8");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/json,application/xml;q=0.9,*/*;q=0.8");
con.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
OutputStream output = con.getOutputStream();
InputStream input = new FileInputStream(new File(FILE_PATH));
writer = new PrintWriter(new OutputStreamWriter(output));
writer.println(boundary);
writer.println("Content-Disposition: form-data; name=\"input1\"");
writer.println();
writer.println("1234");
writer.flush();
writer.println(boundary);
writer.println("Content-Disposition: form-data; name=\"input1\"");
writer.println();
writer.println("asdf");
writer.flush();
writer.println(boundary);
writer.println("Content-Disposition: form-data; name=\"file1\"; filename=\"clicknpoint.png\"");
writer.println("Content-Type: image/png");
writer.println();
writer.flush();
int length = 0;
byte[] buffer = new byte[1024];
for(length = 0; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
writer.flush();
input.close();
writer.println();
writer.println(boundary + "--");
writer.flush();
input = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String cur = null;
StringBuffer buf = new StringBuffer();
while((cur = reader.readLine()) != null) {
buf.append(cur);
}
The server doesn't recognizes the params in the request. I checked with wireshark, they are there but the IP header checksum is 0x0000. I think that's problem.
Any idea where this is coming from?

The only thing I can see that might be incorrect is your use of println to generate HTTP content. Depending on your platform, the println's may be outputting a single LF character or may be outputting a CR LF pair.
HTTP definitely requires CRLF after every header. See http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4. Right now your code is at risk of violating the HTTP protocol. It might work on some platforms but not on others. Also some servers might tolerate the violation of the protocol, but others might not. The least invasive change might be changing every println to print and explicitly adding the CRLF.
writer.print("Content-Disposition: form-data; name=\"input1\"\r\n");
This may have nothing to do with your problem, but until you tighten up your conformance with HTTP your code will be at risk of failing.

As you have already realized, you are looking at two independent issues: Wireshark is showing an invalid checksum, and your server is not correctly receiving data.
Issue 1: Wireshark is showing an invalid checksum
I realize you've already figured out that this is because of TCP checksum offloading. For posterity, I'll repeat what you already know.
On modern computers, the OS does not calculate a checksum for each TCP packet it sends out. Instead, the checksum is calculated in hardware on the Network adapter. Since Wireshark is intercepting packets before they reach the network adapter, it sees the wrong checksum. To confirm that offloading is in fact the reason you are seeing invalid checksums in Wireshark, you can temporarily turn it off. For Mac OS X, use these commands in Terminal:
sudo sysctl -w net.link.ether.inet.apple_hwcksum_tx=0
sudo sysctl -w net.link.ether.inet.apple_hwcksum_rx=0
To re-enable after testing, just replace those 0s with 1s. To disable offloading on other operating systems, here in the section Step 3. If possible, disable TCP and UDP checksum offloading and TCP segmentation offloading.
Issue 2: Server is not correctly receiving data
URLConnection objects are not reusable. And calling flush() or write() on the output stream of a URLConnection immediately sends your request. After that, you must create a new URLConnection. Make sure that all your data has been added to the output stream before calling either of these methods. A really excellent discussion of the consequences of manipulating the I/O streams of URLConnections can be found here:
http://www.javaworld.com/javaworld/jw-03-2001/jw-0323-traps.html?page=4

How to force browser to download file?

Everything works fine, but only if file is small, about 1MB, when I tried it with bigger files, like 20MB my browser display it, instead of force to download, I tried many headers so far, now my code looks:
PrintWriter out = response.getWriter();
String fileName = request.getParameter("filename");
File f= new File(fileName);
InputStream in = new FileInputStream(f);
BufferedInputStream bin = new BufferedInputStream(in);
DataInputStream din = new DataInputStream(bin);
while(din.available() > 0){
out.print(din.readLine());
out.print("\n");
}
response.setContentType("application/force-download");
response.setContentLength((int)f.length());
response.setHeader("Content-Transfer-Encoding", "binary");
response.setHeader("Content-Disposition","attachment; filename=\"" + "xxx\"");//fileName);
in.close();
bin.close();
din.close();

You are setting the response headers after writing the contents of the file to the output stream. This is quite late in the response lifecycle to be setting headers. The correct sequence of operations should be to set the headers first, and then write the contents of the file to the servlet's outputstream.
Therefore, your method should be written as follows (this won't compile as it is a mere representation):
response.setContentType("application/force-download");
response.setContentLength((int)f.length());
//response.setContentLength(-1);
response.setHeader("Content-Transfer-Encoding", "binary");
response.setHeader("Content-Disposition","attachment; filename=\"" + "xxx\"");//fileName);
...
...
File f= new File(fileName);
InputStream in = new FileInputStream(f);
BufferedInputStream bin = new BufferedInputStream(in);
DataInputStream din = new DataInputStream(bin);
while(din.available() > 0){
out.print(din.readLine());
out.print("\n");
}
The reason for the failure is that it is possible for the actual headers sent by the servlet would be different from what you are intending to send. After all, if the servlet container does not know what headers (which appear before the body in the HTTP response), then it may set appropriate headers to ensure that the response is valid; setting the headers after the file has been written is therefore futile and redundant as the container might have already set the headers. You could confirm this by looking at the network traffic using Wireshark or a HTTP debugging proxy like Fiddler or WebScarab.
You may also refer to the Java EE API documentation for ServletResponse.setContentType to understand this behavior:
Sets the content type of the response being sent to the client, if the response has not been committed yet. The given content type may include a character encoding specification, for example, text/html;charset=UTF-8. The response's character encoding is only set from the given content type if this method is called before getWriter is called.
This method may be called repeatedly to change content type and character encoding. This method has no effect if called after the response has been committed.
...

Set content-type and other headers before you write the file out. For small files the content is buffered, and the browser gets the headers first. For big ones the data come first.

This is from a php script which solves the problem perfectly with every browser I've tested (FF since 3.5, IE8+, Chrome)
header("Content-Disposition: attachment; filename=\"".$fname_local."\"");
header("Content-Type: application/force-download");
header("Content-Transfer-Encoding: binary");
header("Content-Length: ".filesize($fname));
So as far as I can see, you're doing everything correctly. Have you checked your browser settings?

Webservice call returns error 500

I have started a small project in Java.
I have to create a client which will send xml to a url as a HTTP POST request.
I try it using java.net.* package (Following is the piece of code) but I am getting error as follows:
java.io.IOException: Server returned HTTP response code: 500 for URL: "target url"
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at newExample.main(newExample.java:36)
My code is as follows:
try {
URL url = new URL("target url");
URLConnection connection = url.openConnection();
if( connection instanceof HttpURLConnection )
((HttpURLConnection)connection).setRequestMethod("POST");
connection.setRequestProperty("Content-Length", Integer.toString(requestXml.length()) );
connection.setRequestProperty("Content-Type","text/xml; charset:ISO-8859-1;");
connection.setDoOutput(true);
connection.connect();
// Create a writer to the url
PrintWriter writer = new PrintWriter(new
OutputStreamWriter(connection.getOutputStream()));
// Get a reader from the url
BufferedReader reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
writer.println();
writer.println(requestXml);
writer.println();
writer.flush();
String line = reader.readLine();
while( line != null ) {
System.out.println( line );
line = reader.readLine();
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Please help with suitable examples or any other ways of doing this.
Point errors/mistakes in above code or other possibilities.
My Web Service is in spring framework
xml to send is in the string format: requestXml

The problem lies in below code
// Get a reader from the url
BufferedReader reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
As the service might not always return you the proper response... as you are calling a service through http, it can be possible that the server itself is not available or the service is not available. So you should always check for the response code before reading response from streams, based on the response code you've to decide whether to read it from inputStream for success response or from errorStream for failure or exception condition.
BufferedReader reader = null;
if(connection.getResponseCode() == 200)
{
reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
}
else
{
reader = new BufferedReader(new
InputStreamReader(connection.getErrorStream()));
}
This would resolve the problem

The problem is inside your server code or the server configuration:
10.5.1 500 Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.
(w3c.org/Protocols)
If the server is under your control (should be, if I look at the URL [before the edit]), then have a look at the server logs.

Well, you should close your streams and connections. Automatic resource maangement from Java 7 or http://projectlombok.org/ can help. However, this is probably not the main problem.
The main problem is that the server-side fails. HTTP code 500 means server-side error. I can't tell you the reason, because I don't know the server side part. Maybe you should look at the log of the server.

I think that your problem is that you are opening the input stream before you have written and closed the output stream. Certainly, the Sun Tutorial does it that way.
If you open the input stream too soon, it is possible that the output stream will be closed automatically, causing the server to see an empty POST request. This could be sufficient to cause it to get confused and send a 500 response.
Even if this is not what is causing the 500 errors, it is a good idea to do things in the order set out in the tutorial. For a start, if you accidentally read the response before you've finished writing the request, you are likely to (at least temporarily) lock up the connection. (In fact, it looks like your code is doing this because you are not closing the writer before reading from the reader.)
A separate issue is that your code does not close the connection in all circumstances, and is therefore liable to leak network connections. If it does this repeatedly, it is likely to lead to more IOExceptions.

If you are calling an External Webservice and passing a JSON in the REST call, check the datatype of the values passed.
Example:
{ "originalReference":"8535064088443985",
"modificationAmount":
{ "amount":"16.0",
"currency":"AUD"
},
"reference":"20170928113425183949",
"merchantAccount":"MOM1"
}
In this example, the value of amount was sent as a string and the webservice call failed with Server returned HTTP response code: 500.
But when the amount: 16.0 was sent, i.e an Integer was passed, the call went through. Though you have referred API documentation while calling such external APIs, small details like this could be missed.