Looking for a bit of help, I have currently written a HTTP server. It currently handles GET requests fine. However, whilst using POST the buffered reader seems to hang. When the request is stopped the rest of the input stream is read via the buffered reader. I have found a few things on google. I have tried changing the CRLF and the protocol version from 1.1 to 1.0 (browsers automatically make requests as 1.1) Any ideas or help would be appreciated. Thanks
I agree with Hans that you should use a standard and well-tested library to do this. However, if you are writing a server to learn about HTTP, here's some info on doing what you want to do.
You really can't use a BufferedReader because it buffers the input and might read too many bytes from the socket. That's why your code is hanging, the BufferedReader is trying to read more bytes than are available on the socket (since the POST data doesn't have an end of line), and it is waiting for more bytes (which will never be available).
The process to simply parse a POST request is to use the InputStream directly
For each line in the header
read a byte at a time until you get a '\r' and then a '\n'
Look for a line that starts with "Content-Length: ", extract the number at the end of that line.
When you get a header line that is empty, you're done with headers.
Now read exactly the # of bytes that came from the Content-Length header.
Now you can write your response.
Wouldn't write my own implementation. Look at the following existing components, if you want:
a HTTP client: Apache HttpClient
a HTTP server implementation: Apache HttpComponents core (as mentioned by Bombe)
This is not safe! But shows how to get the POST data during an Input Stream after the initial HTTP Headers.
This also only works for POST data coming in as "example=true&bad=false" etc.
private HashMap hashMap = new HashMap();
private StringBuffer buff = new StringBuffer();
private int c = 0;
private String[] post; public PostInputStream(InputStream in) {
try {
//Initalizes avaliable buff
if (in.available() != 0) {
this.buff.appendCodePoint((this.c = in.read()));
while (0 != in.available()) {
//Console.output(buff.toString());
buff.appendCodePoint((this.c = in.read()));
}
this.post = buff.toString().split("&");
for (int i = 0; i < this.post.length; i++) {
String[] n = this.post[i].split("=");
if (n.length == 2) {
hashMap.put(URLDecoder.decode(n[0], "UTF-8"), URLDecoder.decode(n[1], "UTF-8"));
} else {
Console.error("Malformed Post Request.");
}
}
} else {
Console.error("No POST Data");
}
} catch (Exception e) {
e.printStackTrace();
}
}
As karoroberts said you have to check the length of the content sent in the POST. But you still can use BufferedReader.
Just have to check for the Content-Length header for the size of it and after finishing reading all the headers you can set a char array of that size and make the reading of the POST content:
char[] buffer = new char[contentLength];
request.read(buffer);
Where request is the BufferedReader.
If you need the POST content in a string, you can use: String.valueOf(buffer);
Note: BufferedReader.read returns an int of the characters readed, so you could check there for inconsistencies with the Content-Length header.
Related
I'm trying to create a simple Java program that create an HTTP request to a HTTP server hosted locally, by using Socket.
This is my code:
try
{
//Create Connection
Socket s = new Socket("localhost",80);
System.out.println("[CONNECTED]");
DataOutputStream out = new DataOutputStream(s.getOutputStream());
DataInputStream in = new DataInputStream(s.getInputStream());
String header = "GET / HTTP/1.1\n"
+"Host:localhost\n\n";
byte[] byteHeader = header.getBytes();
out.write(byteHeader,0,header.length());
String res = "";
/////////////READ PROCESS/////////////
byte[] buf = new byte[in.available()];
in.readFully(buf);
System.out.println("\t[READ PROCESS]");
System.out.println("\t\tbuff length->"+buf.length);
for(byte b : buf)
{
res += (char) b;
}
System.out.println("\t[/READ PROCESS]");
/////////////END READ PROCESS/////////////
System.out.println("[RES]");
System.out.println(res);
System.out.println("[CONN CLOSE]");
in.close();
out.close();
s.close();
}catch(Exception e)
{
e.printStackTrace();
}
But by when I run it the Server reponse with a '400 Bad request error'.
What is the problem? Maybe some HTTP headers to add but I don't know which one to add.
There are a couple of issues with your request:
String header = "GET / HTTP/1.1\n"
+ "Host:localhost\n\n";
The line break to be used must be Carriage-Return/Newline, i.e. you should change that to
String header = "GET / HTTP/1.1\r\n"
+ "Host:localhost\r\n\r\n";
Next problem comes when you write the data to the OutputStream:
byte[] byteHeader = header.getBytes();
out.write(byteHeader,0,header.length());
The call of readBytes without the specification of a charset uses the system's charset which might be a different than the one that is needed here, better use getBytes("8859_1"). When writing to the stream, you use header.length() which might be different from the length of the resulting byte-array if the charset being used leads to the conversion of one character into multiple bytes (e.g. with UTF-8 as encoding). Better use byteHeader.length.
out.write(byteHeader,0,header.length());
String res = "";
/////////////READ PROCESS/////////////
byte[] buf = new byte[in.available()];
After sending the header data you should do a flush on the OutputStream to make sure that no internal buffer in the streams being used prevents the data to actually be sent to the server.
in.available() only returns the number of bytes you can read from the InputStream without blocking. It's not the length of the data being returned from the server. As a simple solution for starters, you can add Connection: close\r\n to your header data and simply read the data you're receiving from the server until it closes the connection:
StringBuffer sb = new StringBuffer();
byte[] buf = new byte[4096];
int read;
while ((read = in.read(buf)) != -1) {
sb.append(new String(buf, 0, read, "8859_1"));
}
String res = sb.toString();
Oh and independent form the topic of doing an HTTP request by your own:
String res = "";
for(byte b : buf)
{
res += (char) b;
}
This is a performance and memory nightmare because Java is actually caching all strings in memory in order to reuse them. So the internal cache gets filled with each result of this concatenation. A response of 100 KB size would mean that at least 5 GB of memory are allocated during that time leading to a lot of garbage collection runs in the process.
Oh, and about the response of the server: This most likely comes from the invalid line breaks being used. The server will regard the whole header including the empty line as a single line and complains about the wrong format of the GET-request due to additional data after the HTTP/1.1.
According to HTTP 1.1:
HTTP/1.1 defines the sequence CR LF as the end-of-line marker for all
protocol elements except the entity-body [...].
So, you'll need all of your request to be ending with \r\n.
I'm trying to read a HTTP request using only Socket and BufferedReader classes in Java. The problem is that I can't reach the body part of the request. The Buffered reader is giving me only the request line and the headers. Here is part of the code:
bufferedReader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String comando = "";
while((msgDoSocket = bufferedReader.readLine()) != null){
//telaOutput.adicionaFim(msgDoSocket);
try {
comando += msgDoSocket + " ";
//System.out.println(comando);
if(msgDoSocket.isEmpty()){
processaInput(comando);
}
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
}
Here is a WireShark capture showing that the POST body is being sent. My program is running on port 15000 and the data is just a string "teste12345". I'm using the app POSTMAN from google chrome to send the requests.
I'm having exactly the same problem described in this thread but the solutions proposed there didn't work. The request still getting up to the last header and no more. Thanks in advance.
Edit: Problem Solved!
Following suggestion proposed on the answer, I changed the reading to:
reader = new DataInputStream(socket.getInputStream());
String comando = "";
while( (dt = reader.readByte()) >= 0){
comando += dt;
//... do the rest of the stuff
}
Reading it as binary made it possible to reach the body part of the request.
I'm far from being a Java guru, but I bet that readLine only returns with results when it found a sequence of \r\n. since your body is not terminated with \r\n the method readLine never returns. try to manually add that character sequence to your body and see what happens, or alternatively, use the raw InputStreamReader to read the body as byte array.
never the less, you can't expect any http body to actually be a string. it can also be a binary sequence which knows nothing about \r\n.
I am new to netty and I followed this example to write a static file server using netty. But whenever the server serves a large js file. It runs into ClosedChannelException.
The following is my code where I write chunkedFile as http response.
When a large js file is being served I get closedChannelException and the raf file is also closed.
Could you help me figure out what I have done wrong here? Also, is there a simple tutorial where I get understand the basic flow of control in netty?
// Write the content.
ChannelFuture writeFuture = null;
try
{
long fileLength = raf.length();
HttpResponse response = new DefaultHttpResponse(
HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
response.setHeader(HttpHeaders.Names.CONTENT_LENGTH, fileLength);
Channel c = ctx.getChannel();
// Write the initial line and the header.
c.write(response);
writeFuture = c.write(new ChunkedFile(raf, 0, fileLength, 8192));
}
finally
{
raf.close();
if (writeFuture != null)
writeFuture.addListener(ChannelFutureListener.CLOSE);
}
}
<
Calling raf.close() in the finally block is wrong as it may not have it written yet. In fact netty will take care to close it after the write is complete.
I am using Apache HTTPClient 4 to connect to twitter's streaming api with default level access. It works perfectly well in the beginning but after a few minutes of retrieving data it bails out with this error:
2012-03-28 16:17:00,040 DEBUG org.apache.http.impl.conn.SingleClientConnManager: Get connection for route HttpRoute[{tls}->http://myproxy:80->https://stream.twitter.com:443]
2012-03-28 16:17:00,040 WARN com.cloudera.flume.core.connector.DirectDriver: Exception in source: TestTwitterSource
java.lang.IllegalStateException: Invalid use of SingleClientConnManager: connection still allocated.
at org.apache.http.impl.conn.SingleClientConnManager.getConnection(SingleClientConnManager.java:216)
Make sure to release the connection before allocating another one.
at org.apache.http.impl.conn.SingleClientConnManager$1.getConnection(SingleClientConnManager.java:190)
I understand why I am facing this issue. I am trying to use this HttpClient in a flume cluster as a flume source. The code looks like this:
public Event next() throws IOException, InterruptedException {
try {
HttpHost target = new HttpHost("stream.twitter.com", 443, "https");
new BasicHttpContext();
HttpPost httpPost = new HttpPost("/1/statuses/filter.json");
StringEntity postEntity = new StringEntity("track=birthday",
"UTF-8");
postEntity.setContentType("application/x-www-form-urlencoded");
httpPost.setEntity(postEntity);
HttpResponse response = httpClient.execute(target, httpPost,
new BasicHttpContext());
BufferedReader reader = new BufferedReader(new InputStreamReader(
response.getEntity().getContent()));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>30000) break;
}
return new EventImpl(buffer.toString().getBytes());
} catch (IOException ie) {
throw ie;
}
}
I am trying to buffer 30,000 characters in the response stream to a StringBuffer and then return this as the data received. I am obviously not closing the connection - but I do not want to close it just yet I guess. Twitter's dev guide talks about this here It reads:
Some HTTP client libraries only return the response body after the
connection has been closed by the server. These clients will not work
for accessing the Streaming API. You must use an HTTP client that will
return response data incrementally. Most robust HTTP client libraries
will provide this functionality. The Apache HttpClient will handle
this use case, for example.
It clearly tells you that HttpClient will return response data incrementally. I've gone through the examples and tutorials, but I haven't found anything that comes close to doing this. If you guys have used a httpclient (if not apache) and read the streaming api of twitter incrementally, please let me know how you achieved this feat. Those who haven't, please feel free to contribute to answers. TIA.
UPDATE
I tried doing this: 1) I moved obtaining stream handle to the open method of the flume source. 2) Using a simple inpustream and reading data into a bytebuffer. So here is what the method body looks like now:
byte[] buffer = new byte[30000];
while (true) {
int count = instream.read(buffer);
if (count == -1)
continue;
else
break;
}
return new EventImpl(buffer);
This works to an extent - I get tweets, they are nicely being written to a destination. The problem is with the instream.read(buffer) return value. Even when there is no data on the stream, and the buffer has default \u0000 bytes and 30,000 of them, so this value is getting written to the destination. So the destination file looks like this.. " tweets..tweets..tweeets.. \u0000\u0000\u0000\u0000\u0000\u0000\u0000...tweets..tweets... ". I understand the count won't return a -1 coz this is a never ending stream, so how do I figure out if the buffer has new content from the read command?
The problem is that your code is leaking connections. Please make sure that no matter what you either close the content stream or abort the request.
InputStream instream = response.getEntity().getContent();
try {
BufferedReader reader = new BufferedReader(
new InputStreamReader(instream));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if (buffer.length()>30000) {
httpPost.abort();
// connection will not be re-used
break;
}
}
return new EventImpl(buffer.toString().getBytes());
} finally {
// if request is not aborted the connection can be re-used
try {
instream.close();
} catch (IOException ex) {
// log or ignore
}
}
It turns out that it was a flume issue. Flume is optimized to transfer events of size 32kb. Anything beyond 32kb, Flume bails out. (The workaround is to tune event size to be greater than 32KB). So, I've changed my code to buffer 20,000 characters at least. It kind of works, but it is not fool proof. This can still fail if the buffer length exceeds 32kb, however, it hasn't failed so far in an hour of testing - I believe it has to do with the fact that Twitter doesn't send a lot of data on its public stream.
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>20000) break;
}
I have a simple program either a server or client not sure which end it is. I open a port and accept messages on it. this works but it crashed last night when I revived a message via GET not POST. As you may have noticed I am very new to networking.
I am getting the transmission ok (the socket stuff) but i cant read the message.
this is the code i currently use.
//read HTTp header until the message size comes in
for(int eight = 0; eight < 8; eight++)
{
message = in.readLine();
LOGGER.fatal(eight);
LOGGER.fatal(message);
if(message.contains("Content-Length"))
{
try {
Pattern pattern = Pattern.compile("\\d+");
Matcher matcher = pattern.matcher(message);
while (matcher.find()) {
sizeInt = Integer.parseInt(matcher.group());
}
}
catch (Exception e)
{
LOGGER.fatal("size not found on header line", e);
//System.exit(-1);
}
}
}
LOGGER.fatal(sizeInt);
LOGGER.fatal("---------------------------");
char[] buffer =new char[sizeInt];
//skip blank line between header and message
message = in.readLine();
//read message
int fullRead = 0;
int thisRead = 0;
do
{
thisRead = in.read(buffer, fullRead, sizeInt - fullRead);
LOGGER.info(fullRead + " of " + sizeInt + " bytes of message read");
fullRead += thisRead;
}while(fullRead != sizeInt);
The problem iv'e got is that the GET method does not seen to have the size of the message in bytes and without that I don't know how the read the message without it hanging.
I'm using java.
Can anyone suggest how I can edit this code to understand a GET message.
A GET request doesn't have any content - there's nothing to read. Any data has to be in the headers / URL.
It looks like you want to implement an HTTP server (it's not a client, because you want to read HTTP request, not make them). If you're "very new to networking", I wouldn't recommend it. Anyway, if you really want to build your own HTTP server there are examples on the Web, for instance this one. Another approach, if you need an HTTP server in your application is to use com.sun.net.httpserver.HttpServer (the link include a minimal example).
HTTP GET does not have any content, and therefor does not have a content-length. Also, a HTTP POST does not necessarily need to specify Content-Length in HTTP/1.1: HTTP Message Length