I'm messing around with HTTP and sockets in Java and was hoping you could shed some light on this:
When my HTTP server written in Java SE 11 does not read the entire request and then responds, the client does not get it or gets an error. Why is that? Is the client unable to read the response before the server has read the entire request? If the call to readBody is executed in the snippet below, this works fine. It also works fine if the response has the Content-Length header and a text body. That is actually more puzzling to me.
My example request is a POST with the data fds. Postman says "Could not get any request" and curl says "curl: (56) Recv failure: Connection reset by peer".
import java.io.*;
import java.net.Socket;
import java.util.*;
class Handler {
public synchronized void read(Socket incoming) {
try (incoming;
OutputStream outputStream = incoming.getOutputStream();
InputStream inputStream = incoming.getInputStream();
PrintWriter pw = new PrintWriter(outputStream)) {
writeRequest(inputStream);
pw.print("HTTP/1.1 200 OK\r\n");
pw.print("\r\n");
pw.flush();
} catch (IOException e) {
System.out.println("ERROR: " + e.getMessage());
e.printStackTrace();
}
}
private void writeRequest(InputStream inputStream) throws IOException {
String verbLine = readLine(inputStream);
Map<String, String> headers = readHeaders(inputStream);
//readBody(inputStream, headers);
}
private void readBody(InputStream inputStream, Map<String, String> headers) throws IOException {
Optional<String> optKey = headers.keySet().stream()
.filter(k -> k.equalsIgnoreCase("Content-Length"))
.findFirst();
if (optKey.isPresent()) {
int contentLength = Integer.parseInt(headers.get(optKey.get()));
byte[] bytes = inputStream.readNBytes(contentLength);
}
}
private Map<String, String> readHeaders(InputStream inputStream) throws IOException {
Map<String, String> headers = new HashMap<>();
while (true) {
String line = readLine(inputStream);
if (line == null || line.isEmpty()) {
return headers;
}
String key = line.split(":")[0].trim();
String value = line.split(":")[1].trim();
headers.put(key, value);
}
}
private String readLine(InputStream inputStream) throws IOException {
byte[] buf = new byte[200];
int offset = 0;
while (true) {
int read = inputStream.read();
if (read == -1) {
return null;
}
buf[offset] = (byte) read;
if (buf[0] == '\n' || (buf[0] == '\r' && buf[1] == '\n')) {
return "";
}
if (buf[offset] == 0x0A) {
int endOfLine = buf[offset - 1] == 0x0D ? offset - 1 : offset;
return new String(buf, 0, endOfLine);
} else {
offset++;
}
}
}
}
If you close a socket at the server while there are still unread data it will result in a connection reset error at the client. This happens here since you don't read the full request. This error will be exposed to the user if the full response from the server was not read yet.
If you send the response with a content-length and then the full body then the client will have read the full response and thus the error will be ignored. If instead you send neither content-length nor use chunked encoding the client will expect the response to end with a proper close of the TCP connection. In this case the connection reset will be propagated to the user since the full response from the server was not (properly) read yet.
Your response needs to have either a Content-Length header or a Transfer-Encoding header - which tells the client how the response will be transmitted and allows it to figure out when all the bytes have been received. Without that it will need to wait for EOF and assume that the response body is terminated by EOF (this for compatibility with HTTP/1.0). It is possible that your client doesn't support that.
It might help to know which client you are using.
Related
In the following code, EntityUtils.toString is going into IOException. When I paste 'EntityUtils.toString(entity)' on eclipse watch window, it showing me the value which is DISABLED
private String triggerRestApiCalls(String url){
HttpClient httpClient = new DefaultHttpClient();
HttpGet getRequest = new HttpGet(url);
getRequest.setHeader(
new BasicHeader("Accept", "application/json"));
try {
HttpResponse response = httpClient.execute(getRequest);
HttpEntity entity = response.getEntity();
String value = EntityUtils.toString(entity);
return value;
} catch (ClientProtocolException e) {
log.debug(e.getCause());
} catch (IOException e) {
log.debug(e.getCause());
}
log.debug("Status Unknown");
return "UNKNOWN";
}
The content value lenght is 8. The string expected is DISABLED, which is exactly of the length. The HTTP status is 200 (OK).
I used curl with same URL.
curl -i -H {Accept: application/json} http://127.0.0.1:9031/test/test.html?someDetails
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=D35C61744F4FB3A47B624FF3D0BEB026; Path=/mics/; Secure; HttpOnly
Content-Type: text/plain;charset=ISO-8859-1
Content-Length: 8
Date: Wed, 26 Nov 2014 13:23:30 GMT
DISABLED.
Any help is appreciated ! Is there any angle to encoding ?
FIRST EDIT
The Stack Trace mentions this.
java.io.IOException: Attempted read from closed stream.
The code that is executed by EntityUtils.toString()
public static String toString(
final HttpEntity entity, final Charset defaultCharset) throws IOException, ParseException {
if (entity == null) {
throw new IllegalArgumentException("HTTP entity may not be null");
}
InputStream instream = entity.getContent();
if (instream == null) {
return null;
}
try {
if (entity.getContentLength() > Integer.MAX_VALUE) {
throw new IllegalArgumentException("HTTP entity too large to be buffered in memory");
}
int i = (int)entity.getContentLength();
if (i < 0) {
i = 4096;
}
ContentType contentType = ContentType.getOrDefault(entity);
Charset charset = contentType.getCharset();
if (charset == null) {
charset = defaultCharset;
}
if (charset == null) {
charset = HTTP.DEF_CONTENT_CHARSET;
}
Reader reader = new InputStreamReader(instream, charset);
CharArrayBuffer buffer = new CharArrayBuffer(i);
char[] tmp = new char[1024];
int l;
while((l = reader.read(tmp)) != -1) {
buffer.append(tmp, 0, l);
}
return buffer.toString();
} finally {
instream.close();
}
}
I have stepped through this and there are not errors However, it closes the stream before it returns.
But the actual value that is returned is CharArrayBuffer which is not linked to the stream. The same code works in some other java file. Strange !! I am using spring.. is there a spring angle to this ?
The HttpEntity can only be read once and it seems something else is intercepting and reading the response so that when you attempt to apply EntityUtils.toString() you get this exception. I can't see why this would be happening, though you did mention there could be a Spring angle so there could be a Spring interceptor applied here.
You could try
String value = httpClient.execute(getRequest, new BasicResponseHandler());
Although from what I can see this should be fairly equivalent to the above code.
As an assignment, I am allowed to use ServerSocket and Socket class only. Also it should be single-threaded as well.
I'm implementing a HTTP proxy server in Java, first it fetches request from client and then pushes to server, and then pushes the response back to the client.
The problem
The problem is, I have successfully get the request, send it to the end-server and get the proper HTTP response. I also can do print out the response in console. But it got stuck when I send the response to clientServer.outputstream. Firefox (requested to use, HTTP 1.0, no keep-alive requested) seems to load forever and nothing shows, and no response Firefox received from my program as well.
What I inspect when debug
Everytime a page start to load (FF request), there are always 2 client sockets. First socket contains null request, and second socket contains proper request. What I expect was that only one proper HTTP request from Firefox. Is that a weird behavior?
example:
/0:0:0:0:0:0:0:1:65194
[null request]
/0:0:0:0:0:0:0:1:65195
GET http://www.microsoft.com/ HTTP/1.0
Host: www.microsoft.com
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: close
Proxy-Connection: close
Cookie: viewkey=lightweight; WT_FPC=id=269eb0e7618962f93a81347585923074:lv=1349229942007:ss=1349229580158; WT_NVR_RU=0=technet|msdn:1=:2=; omniID=c736269c_f430_4e9b_a42a_23a0c965c60a; MUID=212A1766CFE761423CD014BDCBE76158&TUID=1; MC1=GUID=08600fba7f5c5f409e67980d8a027593&HASH=ba0f&LV=20129&V=4&LU=1347643534618; A=I&I=AxUFAAAAAADGBwAA8ezRtqBBHjk3++mP1Bwj9w!!&V=4&CS=119EQ5002j10100; msdn=L=en-US
Code
ServerSocket serverSocket;
try {
serverSocket = new ServerSocket(60000);
while (true) {
clientSocket = serverSocket.accept();
[...]
// Extract request, and push to end-server
// Fetch response from end-server to client, using flush() already
// Close all input, output
// Close all sockets
} catch {[...]}
Any help is welcomed, thank you!
Full code as requested, I use PrintWriter, but before that using Byte makes no difference (not care efficiency)
import java.io.*;
import java.net.*;
import java.util.*;
public class Proxy {
static String separator = System.getProperty("line.separator");
public static void main(String args[]) {
//int port = Integer.parseInt(args[0]);
start(60000);
}
public static void start(int port) {
ServerSocket serverSocket;
try {
serverSocket = new ServerSocket(port);
Socket clientSocket = null;
while (true) {
clientSocket = serverSocket.accept();
System.out.println(clientSocket.getRemoteSocketAddress() + "\n" + clientSocket.getLocalSocketAddress() + "\n" + clientSocket.getInetAddress());
BufferedReader inStreamFromClient = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String inLine;
Vector<String> clientRequestHeader = new Vector<String>();
String rawRequest = "";
while ((inLine = inStreamFromClient.readLine()) != null) {
if (!inLine.isEmpty()) {
clientRequestHeader.add(inLine);
rawRequest = rawRequest.concat(inLine + separator);
} else break;
}
while ((inLine = inStreamFromClient.readLine()) != null)
rawRequest = rawRequest.concat(inLine + separator);
System.out.println(rawRequest);
if (!rawRequest.isEmpty()) {
handleRequest(clientSocket, clientRequestHeader, rawRequest);
} else {
//clientSocket.close();
// Not sure how to handle null request
}
}
} catch (Exception e) {e.printStackTrace();}
}
public static void handleRequest(Socket clientSocket, Vector<String> clientRequestHeader, String rawRequest) {
HTTPRequest request = new HTTPRequest(clientRequestHeader, rawRequest);
try {
//System.out.println(rawRequest);
// Send request to end-server
Socket endServerSocket = new Socket(request.getHost(), 80);
PrintWriter outStreamToEndServer = new PrintWriter(new BufferedWriter(new OutputStreamWriter(endServerSocket.getOutputStream())));
BufferedReader stringReader = new BufferedReader(new StringReader(rawRequest));
String inLine;
while ((inLine = stringReader.readLine())!= null) {
outStreamToEndServer.println(inLine);
}
outStreamToEndServer.println();
outStreamToEndServer.flush();
// Read response header from end-server
String responseHeader = "";
BufferedReader inStreamFromEndServer = new BufferedReader(new InputStreamReader(endServerSocket.getInputStream()));
while (!(inLine = inStreamFromEndServer.readLine()).isEmpty()) {
responseHeader = responseHeader.concat(inLine + separator);
}
// Send response header to client
PrintWriter outStreamToClient = new PrintWriter(new BufferedWriter(new OutputStreamWriter(clientSocket.getOutputStream())));
outStreamToClient.println(responseHeader);
outStreamToClient.flush();
// Send response body to client
String responseBody = "";
while ((inLine = inStreamFromEndServer.readLine()) != null) {
responseBody = responseBody.concat(inLine + separator);
}
outStreamToClient.println(responseBody);
outStreamToClient.flush();
endServerSocket.shutdownInput();
clientSocket.shutdownOutput();
clientSocket.close();
endServerSocket.close();
//endServerSocket = null;
} catch (Exception e) {
e.printStackTrace();
}
}
}
first you should not use PrintWriter to transfer the Data, because the HTTP protocol isn't a pure text protocol the body can contain some raw data like images.
Replace your response transfer code with the code below.
InputStream in = endServerSocket.getInputStream();
OutputStream out = clientSocket.getOutputStream();
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1)
{
out.write(buffer, 0, bytesRead);
}
in.close();
out.close();
Second point, you add always as line break the
static String separator = System.getProperty("line.separator");
This is the System specific line seperator. HTTP defines for the HTTP header and for the http header and body separation the ctrl line break charaters, so change this.
static String separator = "\r\n";
With this changes you will get your response to your browser.
Last Point you should change your client request read code also, because it will not always work if you want POST some data. Sometimes this data will transfered as raw data, by example file uploads.
Good Luck
I need to know what is the problem in the following code
public class NewClass {
public static void main(String[] args) {
try {
while (true) {
ServerSocket ss = new ServerSocket(7777);
Socket c = ss.accept();
BufferedReader reader = new BufferedReader(new InputStreamReader(c.getInputStream()));
DataOutputStream writer = new DataOutputStream(c.getOutputStream());
String temp;
// read browser Request
while ((temp = reader.readLine()) != null) {
System.out.println(temp);
}
// send basic authentication request
String response = "WWW-Authenticate: Basic realm=\"test\"\n";
respons += "HTTP/1.1 401 Authorization Required\n";
writer.writeBytes(response );
writer.flush();
// receive browser response
while ((temp = reader.readLine()) != null) {
System.out.println(temp);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
when i request http://localhost:7777 from browser, the authentication dialog does not appear
why ????
also i'm tried to send this
String response = "HTTP/1.1 401 Authorization Required\n";
response += "WWW-Authenticate: Basic realm=\"test\"\n";
also i sent full server response and without benefits
The reader.readLine() will not return null until it reaches the EOF so you are blocked reading in the first loop and never send anything to the browser.
In the first loop look for an empty string or null.
while ((temp = reader.readLine()) != null) {
if( temp.length() == 0 ) break;
System.out.println(temp);
}
It works fine
String response = "WWW-Authenticate: Basic realm=\"test\"\r\n";
respons += "HTTP/1.1 401 Authorization Required\r\n\r\n";
writer.writeBytes(response );
writer.flush();
but one one problem faced me, how to tell server to wait until browser send basic http authentication.
in my case browser request a new http request.
Thanks
I use HttpURLConnection to do HTTP POST but I dont always get back the full response. I wanted to debug the problem, but when I step through each line it worked. I thought it must be a timing issue so I added Thread.sleep and it really made my code work, but this is only a temporary workaround. I wonder why is this happening and how to solve. Here is my code:
public static InputStream doPOST(String input, String inputMimeType, String url, Map<String, String> httpHeaders, String expectedMimeType) throws MalformedURLException, IOException {
URL u = new URL(url);
URLConnection c = u.openConnection();
InputStream in = null;
String mediaType = null;
if (c instanceof HttpURLConnection) {
//c.setConnectTimeout(1000000);
//c.setReadTimeout(1000000);
HttpURLConnection h = (HttpURLConnection)c;
h.setRequestMethod("POST");
//h.setChunkedStreamingMode(-1);
setAccept(h, expectedMimeType);
h.setRequestProperty("Content-Type", inputMimeType);
for(String key: httpHeaders.keySet()) {
h.setRequestProperty(key, httpHeaders.get(key));
if (logger.isDebugEnabled()) {
logger.debug("Request property key : " + key + " / value : " + httpHeaders.get(key));
}
}
h.setDoOutput(true);
h.connect();
OutputStream out = h.getOutputStream();
out.write(input.getBytes());
out.close();
mediaType = h.getContentType();
logger.debug(" ------------------ sleep ------------------ START");
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
logger.debug(" ------------------ sleep ------------------ END");
if (h.getResponseCode() < 400) {
in = h.getInputStream();
} else {
in = h.getErrorStream();
}
}
return in;
}
later I do the following to read the input stream
ByteArrayOutputStream bos = new ByteArrayOutputStream();
while (is.available() > 0) {
bos.write(is.read());
}
is.close();
//is.read(bytes);
if (logger.isDebugEnabled()) {
logger.debug(" Response lenght is : " + is.available());
//logger.debug("RAW response is " + new String(bytes));
logger.debug("RAW response is " + new String(bos.toByteArray()));
}
It genearates the following HTTP headers
POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39",oauth_nonce="YnDb5eepuLm%2Fbs",oauth_signature="dbN%2FWeWs2G00mk%2BX6uIi3thJxlM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276524919", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: keep-alive
Content-Length: 1107
In other posts it was suggested to turn off keep-alive by using the
http.keepAlive=false
system property, I tried that and the headers changed to
POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39", oauth_nonce="Eaiezrj6X4Ttt0", oauth_signature="ND9fAdZMqbYPR2j%2FXUCZmI90rSI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276526608", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: close
Content-Length: 1107
the Connection header is "close" but I still cannot read the whole response. Any idea what do I do wrong?
I think your problem is in this line:
while (is.available() > 0) {
According to the javadoc, available does not block and wait until all data is available, so you might get the first packet and then it will return false. The proper way to read from an InputStream is like this:
int len;
byte[] buffer = new byte[4096];
while (-1 != (len = in.read(buffer))) {
bos.write(buffer, 0, len);
}
Read will return -1 when there nothing left in the inputstream or the connection is closed, and it will block and wait for the network while doing so. Reading arrays is also much more performant than using single bytes.
Maybe I missed it, but what's the datatype of "input" in your code? Something that's strange about InputStreams in general is that the read( ... ) methods tend to block until data is available, then return only that data. You'll actually need to keep reading from your InputStream and appending to a ByteArrayInputStream or some other structure until you explicitly force a EOFException.
If you are reading the whole Message at once you can compare the isr.available() to the expected content lenght. This is how I did it:
public byte[] readData(HttpURLConnection conn)
throws IOException, InterruptedException {
String _connlen = conn.getHeaderField("Content-Length");
int connlen = Integer.parseInt(_connlen);
InputStream isr = null;
byte[] bytes = new byte[connlen];
try {
isr = conn.getInputStream();
//security count that it doesn't begin to hang
int maxcounter = 0;
//wait till all data is avalibal, max 5sec
while((isr.available() != connlen) && (maxcounter < 5000)){
Thread.sleep(1);
maxcounter++;
}
//Throw if not all data could be read
if(maxcounter >= 5000)
throw new IllegalAccessError();
//read the data
if(isr.read(bytes, 0, connlen) < 0)
throw new IllegalAccessError();
} finally {
if (isr != null)
isr.close();
}
return bytes;
}
Is there any way to get raw response http header?
The getHeaderField() method doesn't work for me, because server spits multiple 'Set-Cookie' and some of them get lost.
The getHeaderField() method doesn't work for me
You're asking this in the context of java.net.URLConnection, is it? No, obtaining the raw HTTP response headers is not possible with URLconnection. You'll need to fall back to low-level Socket programming. Here's an SSCCE, just copy'n'paste'n'run it.
package com.stackoverflow.q2307291;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.net.Socket;
public class Test {
public static void main(String[] args) throws IOException {
String hostname = "stackoverflow.com";
int port = 80;
Socket socket = null;
PrintWriter writer = null;
BufferedReader reader = null;
try {
socket = new Socket(hostname, port);
writer = new PrintWriter(new OutputStreamWriter(socket.getOutputStream()));
writer.println("GET / HTTP/1.1");
writer.println("Host: " + hostname);
writer.println("Accept: */*");
writer.println("User-Agent: Java"); // Be honest.
writer.println(""); // Important, else the server will expect that there's more into the request.
writer.flush();
reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
for (String line; (line = reader.readLine()) != null;) {
if (line.isEmpty()) break; // Stop when headers are completed. We're not interested in all the HTML.
System.out.println(line);
}
} finally {
if (reader != null) try { reader.close(); } catch (IOException logOrIgnore) {}
if (writer != null) { writer.close(); }
if (socket != null) try { socket.close(); } catch (IOException logOrIgnore) {}
}
}
}
To avoid SO being overloaded by everyone trying this snippet, here's how the output will look like:
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Expires: Sun, 21 Feb 2010 20:39:08 GMT
Server: Microsoft-IIS/7.5
Date: Sun, 21 Feb 2010 20:39:07 GMT
Connection: close
Content-Length: 208969
To learn more about sending HTTP requests the low-level way, read the HTTP specification.
However, you probably want to make use of getHeaderFields() method instead to retrieve a header with multiple values. The getHeaderField() namely only returns the last value, as per the linked API doc.
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
Not exactly 'raw' but concise:
for (Map.Entry<String, List<String>> k : myHttpURLConnection.getHeaderFields().entrySet()) {
System.out.println(k.toString());
}
IF you worry that some of the headers are getting lost use:
for (Map.Entry<String, List<String>> k : myHttpURLConnection.getHeaderFields().entrySet()) {
for (String v : k.getValue()){
System.out.println(k.getKey() + ":" + v);
}
}
PS: Better late than never. :)
The easy way is to use the getHeaderFields() method of URLConnection. Here is some code that does something equivalent.
static String[] getHeaders(HttpURLConnection con, String header) {
List<String> values = new ArrayList<String>();
int idx = (con.getHeaderFieldKey(0) == null) ? 1 : 0;
while (true) {
String key = con.getHeaderFieldKey(idx);
if (key == null)
break;
if (header.equalsIgnoreCase(key))
values.add(con.getHeaderField(idx));
++idx;
}
return values.toArray(new String[values.size()]);
}
Late to the party, but here's the simplest solution. Just implement CookieStore. (or use the default implementation and let it take care of adding the cookies to your subsequent calls.)
http://docs.oracle.com/javase/7/docs/api/java/net/CookieStore.html
Set your cookie store as the default cookie manager
CookieManager cookieManager = new CookieManager(new MyCookieStore(), CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(cookieManager);
And every new cookie will appear to you in add() in your CookieStore. I had the same problem with params being overwritten by having the same name "Set-Cookie" in a single request, and now I get both the cookie and the sessionId.