We Keep Coding

Unsuccessful in trying to reuse Java client socket

I have a software driver which communicates with a third-party controller; I have an API for using the latter but no visibility of its source code, and the supplier is not co-operative in trying to improve things!
The situation is as follows.
To send a request to the controller, I send an XML packet as the content of an HTTP POST to a servlet, which then sends me the response. The original code, implemented by a previous developer, works stably using java.net.Socket. However, our driver is implemented such that a new socket is created for EVERY request sent and, if the driver gets busy, the third-party controller struggles to keep up in terms of socket handling. In fact, their support guy said to me: "You really need to leave 5 seconds between each request...". This simply isn't commercially acceptable.
To improve performance, I wanted to try leaving our end of the socket open and reusing the socket pretty much indefinitely (given that connections can drop unexpectedly of course, but that's the least of my concerns and is manageable). However, whatever I seem to do, the effect is that if I use Comms.getSocket(false), a new socket is created for each request and everything works OK but bottlenecks when busy. If I use Comms.getSocket(true), the following happens:
Controller is sent first request
Controller responds to first request
Controller is sent second request (maybe 5 seconds later)
Controller never responds to second request or anything after it
postRequest() keeps getting called: for the first 12 seconds, the console outputs "Input shut down ? false" but, after that, the code no longer reaches there and doesn't get past the bw.write() and bw.flush() calls.
The controller allows both HTTP 1.0 and 1.1 but their docs say zilch about keep-alive. I've tried both and the code below shows that I've added Keep-Alive headers as well but the controller, as server, I'm guessing is ignoring them -- I don't think I have any way of knowing, do I ? When in HTTP 1.0 mode, the controller certainly returns a "Connection: close" but doesn't do that in HTTP 1.1 mode.
The likelihood is then that the server side is insisting on a "one socket per request" approach.
However, I wondered if I might be doing anything wrong (or missing something) in the following code to achieve what I want:
private String postRequest() throws IOException {
String resp = null;
String logMsg;
StringBuilder sb = new StringBuilder();
StringBuilder sbWrite = new StringBuilder();
Comms comms = getComms();
Socket socket = comms.getSocket(true);
BufferedReader br = comms.getReader();
BufferedWriter bw = comms.getWriter();
if (null != socket) {
System.out.println("Socket closed ? " + socket.isClosed());
System.out.println("Socket bound ? " + socket.isBound());
System.out.println("Socket connected ? " + socket.isConnected());
// Write the request
sbWrite
.append("POST /servlet/receiverServlet HTTP/1.1\r\n")
.append("Host: 192.168.200.100\r\n")
.append("Connection: Keep-Alive\r\n")
.append("Keep-Alive: timeout=10\r\n")
.append("Content-Type: text/xml\r\n")
.append("Content-Length: " + requestString.length() + "\r\n\r\n")
.append(requestString);
System.out.println("Writing:\n" + sbWrite.toString());
bw.write(sbWrite.toString());
bw.flush();
// Read the response
System.out.println("Input shut down ? " + socket.isInputShutdown());
String line;
boolean flag = false;
while ((line = br.readLine()) != null) {
System.out.println("Line: <" + line + ">");
if (flag) sb.append(line);
if (line.isEmpty()) flag = true;
}
resp = sb.toString();
}
else {
System.out.println("Socket not available");
}
return resp; // Another method will parse the response
}
To ease testing, I provide the socket using an extra Comms helper class and a method called getSocket(boolean reuse) where I can choose to always create a new socket or reuse the one that Comms creates for me, as follows:
public Comms(String ip, int port) {
this.ip = ip;
this.port = port;
initSocket();
}
private void initSocket() {
try {
socket = new Socket(ip, port);
socket.setKeepAlive(true);
socket.setPerformancePreferences(1, 0, 0);
socket.setReuseAddress(true);
bw = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), StandardCharsets.UTF_8));
br = new BufferedReader(new InputStreamReader(socket.getInputStream(), StandardCharsets.UTF_8));
System.out.println("### CREATED NEW SOCKET");
}
catch (UnknownHostException uhe) {
System.out.println("### UNKNOWN HOST FOR SOCKET");
}
catch (IOException ioe) {
System.out.println("### SOCKET I/O EXCEPTION");
}
}
public BufferedReader getReader() { return br; }
public BufferedWriter getWriter() { return bw; }
public Socket getSocket(boolean reuse) {
if (! reuse) initSocket();
return socket;
}
Can anyone help ?

If we assume that keep-alive thing is working as expected, I think the line while ((line = br.readLine()) != null) is a faulty one, as this is kind of infinity loop.
readline() returns null when there is no more data to read, e.g. a EOF, or when server/client closes the connection, that will break-down your reusing socket solution, since an open stream will never cause a null to a readLine() call, but blocking.
You need to fix the alg about reading a response (why not using implemented http client?), checking content-length, and when read the amount of required data from body, go for next loop by keeping the socket alive.
After that setting flag to true, you have to know what kind of data should be read(considering mime/content-type), besides that, the length of data, so reading data using readLine() may not be a good practice here.
Also make sure server allow for persistence connection, by checking if it respects it by responsing the same connection:keep-alive header.

Why browser received nothing while curl does? (When sending HTTP header response)

I wrote Java program that uses Sockets in order to receive the http request from the client (please NOTE: Everything happens in the localhost) and my program using the Socket library outputs the header response and the html body if GET has been chosen by the client. So let me put in an example:
I run my server at port 12345 and my source file consists of the java file and the example html file.
Client, using the web browser enters: http://localhost:12345/path-to-source/File.html
Server gets input via InputStream: GET path-to-file/File.html HTTP/1.1
So it responds via OutputStream the HTTP status (200 OK), the other fields like Content-type, content-length and then the body (html source code).
For some reason, it outputs nothing to the browser. On the other hand, when I use terminal and type:
curl -s -I -X GET localhost:12345/path-to-file/File.html
It actually displays to the terminal to the client the correct output:
HTTP/1.1 200 OK
Host: localhost:12345
User-Agent: curl/7.51.0
Accept: */*
Content-Type: */*
Content-Length: 18900
But for some reason, the Accept is / event though I have expected text/html
So to be clear, my desired output is:
HTTP/1.1 200 OK
Accept: text/html
Content-Type: text/html
Content-Length: 18900
Content: .....
Can you look at my code and spot what am I doing wrong here? Also, I don't want to use other libraries as I am trying to understand how socket works and the low level:
public class WebServerMain {/** This method prints to the output stream (to the client) the necessary fields, like Content type, Content length and etc.
It only serves GET and HEAD requests and if the file exists (200). **/
public static void printToClient(int length,BufferedReader in, PrintWriter out) {
String request;
String response;
try {
while ((request = in.readLine()) != null) {
out.println(request);
if (request.split(" ")[0].equals("Accept:")) {
response = request.split(" ")[1].split(",")[0];
out.print("Content-Type: " + response + "\r\n");
out.print("Content-Length: " + length + "\r\n");
break;
}
}
} catch(IOException e){
System.err.println("Usage: java WebServerMain <document_root> <port>");
}
}
public static void main(String args[]) {
try {
String cd = "/"+ args[0]; // This is the string that user chooses to pick the directory.
int port;
port = Integer.parseInt(args[1]);
int length = 0; // In order to find the length of the content.
String request;
String response;
final String dir = System.getProperty("user.dir");
System.out.println("current dir = " + dir+cd);
// Create a ServerSocket to listen on that port.
ServerSocket ss = new ServerSocket(port);
// Now enter an infinite loop, waiting for & handling connections.
for (;;) {
// Wait for a client to connect. The method will block;
// when it returns the socket will be connected to the client
Socket socket = ss.accept();
// Get input and outputstreams to talk to the client
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter out = new PrintWriter(socket.getOutputStream());
String userInput = in.readLine();
System.out.println(userInput);
System.out.println("Final dir is " + dir+userInput.split(" ")[1]);
String str;
String content = "";
if(!userInput.split(" ")[0].equals("GET") && !userInput.split(" ")[0].equals("HEAD")){
out.println("HTTP/1.1 501 Not Implemented");
} // Checking, if the user chose correct options, GET or HEAD. If neither, we shall print 501 status code.
else {
try {
BufferedReader readContent = new BufferedReader(new FileReader(dir+cd+userInput.split(" ")[1]));
while ((str = readContent.readLine()) != null) {
content += str;
} // If the file exists, we are reading its content to the string called content. String str is a temporary string.
length = content.length();
out.println(userInput.split(" ")[2] + " 200 OK");
} catch (FileNotFoundException f) {
out.println(userInput.split(" ")[2] + " 404 Not Found");
out.close(); // Flush and close the output stream
in.close(); // Close the input stream
socket.close(); // Close the socket itself
}
if (userInput.split(" ")[0].equals("GET")) {
printToClient(length, in, out);
out.println(content);
} else if (userInput.split(" ")[0].equals("HEAD")) {
printToClient(length, in, out);
}
}
// Close socket, breaking the connection to the client, and
// closing the input and output streams
out.close(); // Flush and close the output stream
in.close(); // Close the input stream
socket.close(); // Close the socket itself
} // Now loop again, waiting for the next connection
}
// If anything goes wrong, print an error message
catch (IOException e) {
System.err.println(e.getMessage());
}
catch (ArrayIndexOutOfBoundsException a){
System.err.println("Usage: java WebServerMain <document_root> <port>");
}
}
}

You can check the headers sent by the web browser using the developer console (right-click -> inspect and then go to the network tab). There you can see what is beeing sent.
Regarding the code you have posted I would restructure two parts:
1) In the loop, after accepting a connection, branch the reading and responding logic to a new thread.
2) Instead of scattering the in.readln() throught the code, read everything in one place and then parse it.
If you go this way, it will be easy to log the whole request you received and see if your assumptions regarding what is reaching the server are correct.

public static void printToClient(...
...
out.print("Content-Type: " + response + "\r\n");
out.print("Content-Length: " + length + "\r\n");
break;
public static void main(String args[]) {
...
printToClient(length, in, out);
out.println(content);
There seems to be no \r\n (i.e. empty line) at the end of the HTTP response header which is required in HTTP/1.x to signal the end of the HTTP header. This means that you send an invalid response.
Apart from that your dealing with the request looks very strange for me:
The client is not required to send an Accept header (although at least the browsers do), yet you require one.
For whatever reason you reflect all request headers up to and including the Accept header back to the client, even if these headers don't make any sense in the response.
You send the status line only with println which ends the line with \n only instead the required \r\n.
Your error responses (404, 501) miss the \r\n at the end of the header top, which is required even if no body follows.

Java Socket not receiving body of http request

I'm trying to read a HTTP request using only Socket and BufferedReader classes in Java. The problem is that I can't reach the body part of the request. The Buffered reader is giving me only the request line and the headers. Here is part of the code:
bufferedReader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String comando = "";
while((msgDoSocket = bufferedReader.readLine()) != null){
//telaOutput.adicionaFim(msgDoSocket);
try {
comando += msgDoSocket + " ";
//System.out.println(comando);
if(msgDoSocket.isEmpty()){
processaInput(comando);
}
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
}
Here is a WireShark capture showing that the POST body is being sent. My program is running on port 15000 and the data is just a string "teste12345". I'm using the app POSTMAN from google chrome to send the requests.
I'm having exactly the same problem described in this thread but the solutions proposed there didn't work. The request still getting up to the last header and no more. Thanks in advance.
Edit: Problem Solved!
Following suggestion proposed on the answer, I changed the reading to:
reader = new DataInputStream(socket.getInputStream());
String comando = "";
while( (dt = reader.readByte()) >= 0){
comando += dt;
//... do the rest of the stuff
}
Reading it as binary made it possible to reach the body part of the request.

I'm far from being a Java guru, but I bet that readLine only returns with results when it found a sequence of \r\n. since your body is not terminated with \r\n the method readLine never returns. try to manually add that character sequence to your body and see what happens, or alternatively, use the raw InputStreamReader to read the body as byte array.
never the less, you can't expect any http body to actually be a string. it can also be a binary sequence which knows nothing about \r\n.

HTTP GET request not working in java when HTTP is 1.1?

so i made a little code that can download 4chan pages. i get the raw HTML page and parse it for my need. the code below was working fine but it suddenly stopped working. when i run it the server does not accept my request it seems its waiting for something more. however i know that HTTP request is as below
GET /ck HTTP/1.1
Host: boards.4chan.org
(extra new line)
if i change this format in anyway i revive "400 bad request" status code. but if i change HTTP/1.1 to 1.0 the server responses in "200 ok" status and i get the whole page. so this makes me thing the error is in the host line since that became mandatory in HTTP/1.1. but still i cannot figure out what exactly need to be changed.
the calling function simply this, to get one whole board
downloadHTMLThread( "ck", -1);
or for a specific thread u just change -1 to that number. for example like for the link below will have like below.
//http://boards.4chan.org/ck/res/3507158
//url.getDefaultPort() is 80
//url.getHost() is boards.4chan.org
//url.getFile() is /ck/res/3507158
downloadHTMLThread( "ck", 3507158);
any advise would be appreciated, thanks
public static final String BOARDS = "boards.4chan.org";
public static final String IMAGES = "images.4chan.org";
public static final String THUMBS = "thumbs.4chan.org";
public static final String RES = "/res/";
public static final String HTTP = "http://";
public static final String SLASH = "/";
public String downloadHTMLThread( String board, int thread) {
BufferedReader reader = null;
PrintWriter out = null;
Socket socket = null;
String str = null;
StringBuilder input = new StringBuilder();
try {
URL url = new URL(HTTP+BOARDS+SLASH+board+(thread==-1?SLASH:RES+thread));
socket = new Socket( url.getHost(), url.getDefaultPort());
reader = new BufferedReader( new InputStreamReader( socket.getInputStream()));
out = new PrintWriter(socket.getOutputStream(), true);
out.println( "GET " +url.getFile()+ " HTTP/1.1");
out.println( "HOST: " + url.getHost());
out.println();
long start = System.currentTimeMillis();
while ((str = reader.readLine()) != null) {
input.append( str).append("\r\n");
}
long end = System.currentTimeMillis();
System.out.println( input);
System.out.println( "\nTime: " +(end-start)+ " milliseconds");
} catch (Exception ex) {
ex.printStackTrace();
input = null;
} finally {
if( reader!=null){
try {
reader.close();
} catch (IOException ioe) {
// nothing to see here
}
}
if( socket!=null){
try {
socket.close();
} catch (IOException ioe) {
// nothing to see here
}
}
if( out!=null){
out.close();
}
}
return input==null? null: input.toString();
}

Try using Apache HttpClient instead of rolling your own:
static String getUriContentsAsString(String uri) throws IOException {
HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(new HttpGet(uri));
return EntityUtils.toString(response.getEntity());
}
If you are doing this to really learn the internals of HTTP client requests, then you might start by playing with curl from the command line. This will let you get all your headers and request body squared away. Then it will be a simple matter of adjusting your request to match what works in curl.

By the code I think that you are sending 'HOST' instead of 'Host'. Since this is a compulsory header in http/1.1, but ignored in http/1.0, that might be the problem.
Anyway, you could use a program to capture the packet sent (i. e. wireshark), just to make sure.
Using println is quite useful, but the line separator appended to the command depends on the system property line.separator. I think (although I'm not sure) that the line separator used in http protocol has to be '\r\n'. If you're capturing the packet, I think it'd be a good idea to check that each line sent ends with '\r\n' (bytes x0D0A) (just in case your os line separator is different)

Use www.4chan.org as the host instead. Since boards.4chan.org is a 302 redirect to www.4chan.org, you won't be able to scrape anything from boards.4chan.org.

HttpsURLConnection Connection Problems

I'm a problem with a HttpsURLConnection that I can't seem to solve. Basically, I'm sending up some info to a server and if some of that data is wrong, the server sends me a 500 response code. However, it also sends a message in the response telling me which bit of data was wrong. The problem is that the message is always empty when I read it in. I think this is because a filenotfound exception always gets thrown before the stream can be read. Am I right? I tried reading the errorstream as well but this is always empty. Here's a snippet:
conn = (HttpsURLConnection) connectURL.openConnection();
conn.setDoOutput(true);
conn.setConnectTimeout(30000);
conn.setReadTimeout(30000);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Length",
Integer.toString(outString.getBytes().length));
DataOutputStream wr = new DataOutputStream(conn
.getOutputStream());
wr.write(outString.getBytes());
wr.flush();
wr.close();
if(conn.getResponseCode>400{
String response = getErrorResponse(conn);
public String getErrorResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
//is = conn.getInputStream();
is = conn.getErrorStream();
// scoop up the reply from the server
int ch;
StringBuffer sb = new StringBuffer();
while ((ch = is.read()) != -1) {
sb.append((char) ch);
}
//System.out.println(sb.toString());
return sb.toString();
// return conferenceId;
}
catch (Exception e){
e.printStackTrace();
}
}

So just to follow up on this, here is how I solved it:
public static String getResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
if(conn.getResponseCode()>=400){
is = conn.getErrorStream();
}
else{
is=conn.getInputStream();
}
...read stream...
}
It seems that calling them like this produced an error stream with a message. Thanks for the suggestions!

Try setting content-type request property to "application/x-www-form-urlencoded"
The same is mentioned on this link:
http://developers.sun.com/mobility/midp/ttips/HTTPPost/
The Content-Length and Content-Type headers are critical because they tell the web server how many bytes of data to expect, and what kind, identified by a MIME type.
In MIDP clients the two most popular MIME types are application/octet-stream, to send raw binary data, and application/x-www-form-urlencoded, to send name-value pairs

Are you in control of the server? In other words, did you write the process that runs on the server and listens to the port you're trying to access?
If you did, then you should also be able to debug it and see why your process returns 404.
If you didn't, then describe your architecture (HTTP server, the component it invokes to respond to your HTTP(S) request, etc) and we'll take it from there.
In the very simplest case, of an HTTP server being an Apache server yielding control to some PHP script, it means that Apache couldn't assign your request to anything. Most likely a Web server misconfiguration. Provide some more details and we'll help you out.