Does a http client return a stream as the response or does it reads the bytes from the stream and returns us the response ?
If it returns a stream , the user will have to read from that stream to get the response right ?
For example - We are using ApacheAsyncHttpClient to get blobs from Azure Storage.
The REST response should only contain the stream from where to read the blob right ? But what I observed is that the client itself is reading all the bytes from the stream before invoking our completed Callback.
And unless the data from the stream is read , the HTTP connection is kept open ?
Edit - We cant use sdk for reasons beyond the scope of this question.
My question is specifically for ApacheAsyncHttpClient.
It reads the entire stream before calling the completed callback. Ideally, what I would want is that it returns me the stream and my application will read from that stream.
Is there a way to do that ?
And are all http clients behave similarly ?
I have a TCP socket connected to a server communicating via JSON. Since it's a stream the packets often merge into one larger through the inputstream, and I've been trying to parse it correctly but haven't found a successful way to read the input.
The input looks something like this
+#$3{"data1":"datahere","moredata":25}2*${"data1":"differentdata","otherdata":001}
This I have parsed to be
{"data1":"datahere","moredata":25}{"data1":"differentdata","otherdata":001}
Using a reader only returns the first json object when it is set to lenient, otherwise it throws an exception. The JSON comes from a socket, and is a result of packet merging. What can be done in a JSON library to parse these separately?
I'm looking to try and make a java server that can accept GET and POST HTTP requests. Now I've managed to get the GET method to work. But I'm not managing to get the POST method to work. My server manages to read the Request Header but doesn't seem to read the body of the message. i.e what was posted. Here is the code:
int port = 1991;
ServerSocket serverSocket = new ServerSocket(port);
System.err.println("The Server is on and listening on port " + port);
System.out.println(" ");
while (true)
{
Socket ClientSocketConnection = serverSocket.accept();
System.err.println("We have established a connection with a client!");
System.out.println(" ");
BufferedReader ServerInput = new BufferedReader(new InputStreamReader(ClientSocketConnection.getInputStream()));
DataOutputStream ServerOutput =new DataOutputStream(ClientSocketConnection.getOutputStream());
String StringInput;
int iCount = 0;
int CountNull = 0;
while ((StringInput = ServerInput.readLine()) != null)
{
System.out.println(StringInput);
}
Now I simply display everything that is sent through the socket. But for some reason I just dont get the requests message body and I know the Body is sent because in chrome I have this:
I'm not sure how to get that "Form Data". Any help would really be appreciated!!
UPDATE:
Here is the problem further narrowed down. From sends the HTTP request fine. With an HTTP POST method we have the request header a \r\n and then the message data. The problem is when my BufferedData variable ServerInput reads in the \r\n (empty line) it stops reading from the ServerInput. Anyway to fix this?
You need to read about the HTTP protocol. You could take a look at HttpServlet api for that.
The purpose of servlets is exactly passing from a socket to an Http protocol. Are you sure you want to do the job again?
I'd highly recommend you take a look at Jetty. It's an embeddable http server which will abstract all of this away for you.
As mentioned in the post the problem I had was that the server I made read the HTTP request header but some how never managed to read the post information being sent to the server via Google chrome.
Now an HTTP post request has the following structure:
request header
\r\n
post information
The reason for me not being able to read the post information was because of the .readLine() function! Once the function reads in a \r\n it assumes that is the end of the message and stops reading the post information being sent and hence the error. To fix this problem I had to use the .read() function instead of the .readLine(). The .read() function reads in every character from the HTTP request which included the post information
I am trying to write a simple mail client in Java with sockets. But I am thinking what is the best way to handle the response from the server? Suppose I have a simple code like the one below:
smtpSock = new Socket(mailHost, SMTP_PORT);
inn = smtpSock.getInputStream();
outt = smtpSock.getOutputStream();
in = new BufferedReader(new InputStreamReader(inn));
out = new PrintWriter(new OutputStreamWriter(outt), true);
out.println("AUTH LOGIN " + user.getID());
String response = in.readLine();
//manipulate the response string to check if user ID exists on server
//if ID exists, proceed to password. Else, quit.
out.println("PASS " + user.getPassword());
response = in.readLine();
//manipulate the response string to check if auth success or not
After I have issued the AUTH LOGIN commmand and given the user ID as the parameter, the server is likely going to give a response. I can get the response to a String. Now, to understand the response and know what to do next, I am thinking that I would manipulate the string by tokenising it and then read the first 3 characters one by one to interpret the response code number. This may sound ok since the response code is only 3 characters.
But then what happens if say the socket is now connecting to a POP3 and is retrieving mails. The response is going to be pretty huge. Is there a good and not too complex method to interpret the response and then put them into a data structure? If parsing the string is the only way, would my idea of tokenising the huge set of String data according to its "textual patterns" a little too naive? Is there a more efficient and faster way of doing this?
POP3 is very different than SMTP, among others it uses different response syntax. SMTP has three letter codes, yes, but I would add that there could be extended status codes too following the simple code. POP3 does not have error codes, only a common negative message. You can parse line by line both of them. POP3 does not preceede each line of the message with a status code of course. If the message is very large then you have to write it into a file while you reveive it. I recommend you to look at the source of Java mail servers, Mireka http://code.google.com/p/mireka/ and Apache James http://james.apache.org/
Let's say I have a java program that makes an HTTP request on a server using HTTP 1.1 and doesn't close the connection. I make one request, and read all data returned from the input stream I have bound to the socket. However, upon making a second request, I get no response from the server (or there's a problem with the stream - it doesn't provide any more input). If I make the requests in order (Request, request, read) it works fine, but (request, read, request, read) doesn't.
Could someone shed some insight onto why this might be happening? (Code snippets follow). No matter what I do, the second read loop's isr_reader.read() only ever returns -1.
try{
connection = new Socket("SomeServer", port);
con_out = connection.getOutputStream();
con_in = connection.getInputStream();
PrintWriter out_writer = new PrintWriter(con_out, false);
out_writer.print("GET http://somesite HTTP/1.1\r\n");
out_writer.print("Host: thehost\r\n");
//out_writer.print("Content-Length: 0\r\n");
out_writer.print("\r\n");
out_writer.flush();
// If we were not interpreting this data as a character stream, we might need to adjust byte ordering here.
InputStreamReader isr_reader = new InputStreamReader(con_in);
char[] streamBuf = new char[8192];
int amountRead;
StringBuilder receivedData = new StringBuilder();
while((amountRead = isr_reader.read(streamBuf)) > 0){
receivedData.append(streamBuf, 0, amountRead);
}
// Response is processed here.
if(connection != null && !connection.isClosed()){
//System.out.println("Connection Still Open...");
out_writer.print("GET http://someSite2\r\n");
out_writer.print("Host: somehost\r\n");
out_writer.print("Connection: close\r\n");
out_writer.print("\r\n");
out_writer.flush();
streamBuf = new char[8192];
amountRead = 0;
receivedData.setLength(0);
while((amountRead = isr_reader.read(streamBuf)) > 0 || amountRead < 1){
if (amountRead > 0)
receivedData.append(streamBuf, 0, amountRead);
}
}
// Process response here
}
Responses to questions:
Yes, I'm receiving chunked responses from the server.
I'm using raw sockets because of an outside restriction.
Apologies for the mess of code - I was rewriting it from memory and seem to have introduced a few bugs.
So the consensus is I have to either do (request, request, read) and let the server close the stream once I hit the end, or, if I do (request, read, request, read) stop before I hit the end of the stream so that the stream isn't closed.
According to your code, the only time you'll even reach the statements dealing with sending the second request is when the server closes the output stream (your input stream) after receiving/responding to the first request.
The reason for that is that your code that is supposed to read only the first response
while((amountRead = isr_reader.read(streamBuf)) > 0) {
receivedData.append(streamBuf, 0, amountRead);
}
will block until the server closes the output stream (i.e., when read returns -1) or until the read timeout on the socket elapses. In the case of the read timeout, an exception will be thrown and you won't even get to sending the second request.
The problem with HTTP responses is that they don't tell you how many bytes to read from the stream until the end of the response. This is not a big deal for HTTP 1.0 responses, because the server simply closes the connection after the response thus enabling you to obtain the response (status line + headers + body) by simply reading everything until the end of the stream.
With HTTP 1.1 persistent connections you can no longer simply read everything until the end of the stream. You first need to read the status line and the headers, line by line, and then, based on the status code and the headers (such as Content-Length) decide how many bytes to read to obtain the response body (if it's present at all). If you do the above properly, your read operations will complete before the connection is closed or a timeout happens, and you will have read exactly the response the server sent. This will enable you to send the next request and then read the second response in exactly the same manner as the first one.
P.S. Request, request, read might be "working" in the sense that your server supports request pipelining and thus, receives and processes both request, and you, as a result, read both responses into one buffer as your "first" response.
P.P.S Make sure your PrintWriter is using the US-ASCII encoding. Otherwise, depending on your system encoding, the request line and headers of your HTTP requests might be malformed (wrong encoding).
Writing a simple http/1.1 client respecting the RFC is not such a difficult task.
To solve the problem of the blocking i/o access where reading a socket in java, you must use java.nio classes.
SocketChannels give the possibility to perform a non-blocking i/o access.
This is necessary to send HTTP request on a persistent connection.
Furthermore, nio classes will give better performances.
My stress test give to following results :
HTTP/1.0 (java.io) -> HTTP/1.0 (java.nio) = +20% faster
HTTP/1.0 (java.io) -> HTTP/1.1 (java.nio with persistent connection) = +110% faster
Make sure you have a Connection: keep-alive in your request. This may be a moot point though.
What kind of response is the server returning? Are you using chunked transfer? If the server doesn't know the size of the response body, it can't provide a Content-Length header and has to close the connection at the end of the response body to indicate to the client that the content has ended. In this case, the keep-alive won't work. If you're generating content on-the-fly with PHP, JSP etc., you can enable output buffering, check the size of the accumulated body, push the Content-Length header and flush the output buffer.
Is there a particular reason you're using raw sockets and not Java's URL Connection or Commons HTTPClient?
HTTP isn't easy to get right. I know Commons HTTP Client can re-use connections like you're trying to do.
If there isn't a specific reason for you using Sockets this is what I would recommend :)
Writing your own correct client HTTP/1.1 implementation is nontrivial; historically most people who I've seen attempt it have got it wrong. Their implementation usually ignores the spec and just does what appears to work with one particular test server - in particular, they usually ignore the requirement to be able to handle chunked responses.
Writing your own HTTP client is probably a bad idea, unless you have some VERY strange requirements.