In the following code, EntityUtils.toString is going into IOException. When I paste 'EntityUtils.toString(entity)' on eclipse watch window, it showing me the value which is DISABLED
private String triggerRestApiCalls(String url){
HttpClient httpClient = new DefaultHttpClient();
HttpGet getRequest = new HttpGet(url);
getRequest.setHeader(
new BasicHeader("Accept", "application/json"));
try {
HttpResponse response = httpClient.execute(getRequest);
HttpEntity entity = response.getEntity();
String value = EntityUtils.toString(entity);
return value;
} catch (ClientProtocolException e) {
log.debug(e.getCause());
} catch (IOException e) {
log.debug(e.getCause());
}
log.debug("Status Unknown");
return "UNKNOWN";
}
The content value lenght is 8. The string expected is DISABLED, which is exactly of the length. The HTTP status is 200 (OK).
I used curl with same URL.
curl -i -H {Accept: application/json} http://127.0.0.1:9031/test/test.html?someDetails
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=D35C61744F4FB3A47B624FF3D0BEB026; Path=/mics/; Secure; HttpOnly
Content-Type: text/plain;charset=ISO-8859-1
Content-Length: 8
Date: Wed, 26 Nov 2014 13:23:30 GMT
DISABLED.
Any help is appreciated ! Is there any angle to encoding ?
FIRST EDIT
The Stack Trace mentions this.
java.io.IOException: Attempted read from closed stream.
The code that is executed by EntityUtils.toString()
public static String toString(
final HttpEntity entity, final Charset defaultCharset) throws IOException, ParseException {
if (entity == null) {
throw new IllegalArgumentException("HTTP entity may not be null");
}
InputStream instream = entity.getContent();
if (instream == null) {
return null;
}
try {
if (entity.getContentLength() > Integer.MAX_VALUE) {
throw new IllegalArgumentException("HTTP entity too large to be buffered in memory");
}
int i = (int)entity.getContentLength();
if (i < 0) {
i = 4096;
}
ContentType contentType = ContentType.getOrDefault(entity);
Charset charset = contentType.getCharset();
if (charset == null) {
charset = defaultCharset;
}
if (charset == null) {
charset = HTTP.DEF_CONTENT_CHARSET;
}
Reader reader = new InputStreamReader(instream, charset);
CharArrayBuffer buffer = new CharArrayBuffer(i);
char[] tmp = new char[1024];
int l;
while((l = reader.read(tmp)) != -1) {
buffer.append(tmp, 0, l);
}
return buffer.toString();
} finally {
instream.close();
}
}
I have stepped through this and there are not errors However, it closes the stream before it returns.
But the actual value that is returned is CharArrayBuffer which is not linked to the stream. The same code works in some other java file. Strange !! I am using spring.. is there a spring angle to this ?
The HttpEntity can only be read once and it seems something else is intercepting and reading the response so that when you attempt to apply EntityUtils.toString() you get this exception. I can't see why this would be happening, though you did mention there could be a Spring angle so there could be a Spring interceptor applied here.
You could try
String value = httpClient.execute(getRequest, new BasicResponseHandler());
Although from what I can see this should be fairly equivalent to the above code.
Related
I'm messing around with HTTP and sockets in Java and was hoping you could shed some light on this:
When my HTTP server written in Java SE 11 does not read the entire request and then responds, the client does not get it or gets an error. Why is that? Is the client unable to read the response before the server has read the entire request? If the call to readBody is executed in the snippet below, this works fine. It also works fine if the response has the Content-Length header and a text body. That is actually more puzzling to me.
My example request is a POST with the data fds. Postman says "Could not get any request" and curl says "curl: (56) Recv failure: Connection reset by peer".
import java.io.*;
import java.net.Socket;
import java.util.*;
class Handler {
public synchronized void read(Socket incoming) {
try (incoming;
OutputStream outputStream = incoming.getOutputStream();
InputStream inputStream = incoming.getInputStream();
PrintWriter pw = new PrintWriter(outputStream)) {
writeRequest(inputStream);
pw.print("HTTP/1.1 200 OK\r\n");
pw.print("\r\n");
pw.flush();
} catch (IOException e) {
System.out.println("ERROR: " + e.getMessage());
e.printStackTrace();
}
}
private void writeRequest(InputStream inputStream) throws IOException {
String verbLine = readLine(inputStream);
Map<String, String> headers = readHeaders(inputStream);
//readBody(inputStream, headers);
}
private void readBody(InputStream inputStream, Map<String, String> headers) throws IOException {
Optional<String> optKey = headers.keySet().stream()
.filter(k -> k.equalsIgnoreCase("Content-Length"))
.findFirst();
if (optKey.isPresent()) {
int contentLength = Integer.parseInt(headers.get(optKey.get()));
byte[] bytes = inputStream.readNBytes(contentLength);
}
}
private Map<String, String> readHeaders(InputStream inputStream) throws IOException {
Map<String, String> headers = new HashMap<>();
while (true) {
String line = readLine(inputStream);
if (line == null || line.isEmpty()) {
return headers;
}
String key = line.split(":")[0].trim();
String value = line.split(":")[1].trim();
headers.put(key, value);
}
}
private String readLine(InputStream inputStream) throws IOException {
byte[] buf = new byte[200];
int offset = 0;
while (true) {
int read = inputStream.read();
if (read == -1) {
return null;
}
buf[offset] = (byte) read;
if (buf[0] == '\n' || (buf[0] == '\r' && buf[1] == '\n')) {
return "";
}
if (buf[offset] == 0x0A) {
int endOfLine = buf[offset - 1] == 0x0D ? offset - 1 : offset;
return new String(buf, 0, endOfLine);
} else {
offset++;
}
}
}
}
If you close a socket at the server while there are still unread data it will result in a connection reset error at the client. This happens here since you don't read the full request. This error will be exposed to the user if the full response from the server was not read yet.
If you send the response with a content-length and then the full body then the client will have read the full response and thus the error will be ignored. If instead you send neither content-length nor use chunked encoding the client will expect the response to end with a proper close of the TCP connection. In this case the connection reset will be propagated to the user since the full response from the server was not (properly) read yet.
Your response needs to have either a Content-Length header or a Transfer-Encoding header - which tells the client how the response will be transmitted and allows it to figure out when all the bytes have been received. Without that it will need to wait for EOF and assume that the response body is terminated by EOF (this for compatibility with HTTP/1.0). It is possible that your client doesn't support that.
It might help to know which client you are using.
I am trying to get the source of a webpage using the following code:
public static String getFile(String sUrl) throws ClientProtocolException, IOException {
DefaultHttpClient httpclient = new DefaultHttpClient();
StringBuilder b = new StringBuilder();
// Prepare a request object
HttpGet httpget = new HttpGet(sUrl);
// Execute the request
HttpResponse response = httpclient.execute(httpget);
// Examine the response status
System.out.println(response.getStatusLine());
//status code should be 200
if (response.getStatusLine().getStatusCode() != 200) {
return null;
}
// Get hold of the response entity
HttpEntity entity = response.getEntity();
// If the response does not enclose an entity, there is no need
// to worry about connection release
if (entity != null) {
InputStream instream = entity.getContent();
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(instream));
// do something useful with the response
String s = reader.readLine();
while (s != null) {
b.append(s);
b.append("\n");
s = reader.readLine();
}
} catch (IOException ex) {
// In case of an IOException the connection will be released
// back to the connection manager automatically
throw ex;
} catch (RuntimeException ex) {
// In case of an unexpected exception you may want to abort
// the HTTP request in order to shut down the underlying
// connection and release it back to the connection manager.
httpget.abort();
throw ex;
} finally {
// Closing the input stream will trigger connection release
instream.close();
}
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
return b.toString();
}
It works fine, but certain symbols like  , - , single quotes etc. are not getting copied correctly.
I try to save the page source as a text/html type into amazon s3 and display it by accessing the page saved in the s3 server.
The symbols that I mentioned above are displayed as � .
Is there any solution for this?
You need to make sure that you are reading the content with the encoding of the page, else your system default encoding would be used (which apparently is not the correct one as you have seen):
BufferedReader reader = new BufferedReader(
new InputStreamReader(instream, entity.getContentEncoding()));
First one need to specify the encoding that the InputStreamReader uses. Your version of the constructor takes the default encoding on your system.
The encoding could be delivered in the headers. It defaults to ISO-8859-1 but (Latin-1) but in reality is Windows-1252 (Windows Latin-1).
String charset = "Windows-1252"; // Can be used as default.
String enc = entity.getContentEncoding(); // Or from Content-Type.
if (enc != null) {
charset = enc;
}
BufferedReader reader = new BufferedReader(
new InputStreamReader(instream, charset));
For HTML entities, apache has:
String s = ...
s = StringEscapeUtils.unescapeHTML4(s);
I'm trying to parse an HTTP request using the apache httpcore components and want to grab the body of the request. It looks like the default DefaultHttpRequestParser doesn't parse the body/entity from its input stream. Is there a class that will do this?
Unfortunately I can't use the entire stack and need to pull the request straight from this input stream.
My parsing code is below. Looking at some of the other answers it appears that the body of the request should be available as an entity. However, every time I try to get at the entity it is null.
Debugging I see that the buffer has read but not used the body and that DefaultHttpRequestParser seems to just read the header. Is there a parse I should be using to parse the entire input?
InputStream is = socket.getInputStream();
HttpTransportMetricsImpl metrics = new HttpTransportMetricsImpl();
SessionInputBufferImpl buf = new SessionInputBufferImpl(metrics, 2048);
buf.bind(is);
DefaultHttpRequestParser reqParser = new DefaultHttpRequestParser(buf);
HttpRequest req = reqParser.parse();
if (req instanceof HttpEntityEnclosingRequest) {
entity = ((HttpEntityEnclosingRequest)query).getEntity();
//... entity is always null
If I read the input stream I end up with:
POST / HTTP/1.1
User-Agent: curl/Q.XX.0 (linux-gnu) libcurl/Q.XX.0 OpenSSL/X.Y.Z zlib/A.B.C.D libidn/E.FF librtmp/G.H
Host: localhost:8088
Accept: */*
Content-Length: 333
Content-Type: multipart/form-data; boundary=----------------------------39203c7982df
------------------------------39203c7982df
Content-Disposition: form-data; name="fileupload"; filename="grun.sh"
Content-Type: application/octet-stream
#!/bin/bash -x
java -classpath lib/antlr-4.4-complete.jar:build/classes org.antlr.v4.runtime.misc.TestRig Typegroup "AHI" -tree
------------------------------39203c7982df--
[Update] Oleg has a good answer, but can I associate the body with the request or do
I now need to pass two things around, the body and the stream? I'll be looking into
I got the following to work, but it's deprecated in the latest release.
...
HttpEntityEnclosingRequest ereq = (HttpEntityEnclosingRequest) req;
#SuppressWarnings("deprecation")
EntityDeserializer ed =
new EntityDeserializer(new LaxContentLengthStrategy());
#SuppressWarnings("deprecation")//ack!
HttpEntity ent = ed.deserialize(buf, req);
ereq.setEntity(ent);
return ereq;
Combining Oleg's solution with the above I ended up with:
HttpEntityEnclosingRequest ereq = (HttpEntityEnclosingRequest) req;
ContentLengthStrategy contentLengthStrategy =
StrictContentLengthStrategy.INSTANCE;
long len = contentLengthStrategy.determineLength(req);
InputStream contentStream = null;
if (len == ContentLengthStrategy.CHUNKED) {
contentStream = new ChunkedInputStream(buf);
} else if (len == ContentLengthStrategy.IDENTITY) {
contentStream = new IdentityInputStream(buf);
} else {
contentStream = new ContentLengthInputStream(buf, len);
}
BasicHttpEntity ent = new BasicHttpEntity();
ent.setContent(contentStream);
ereq.setEntity(ent);
return ereq;
InputStream is = socket.getInputStream();
HttpTransportMetricsImpl metrics = new HttpTransportMetricsImpl();
SessionInputBufferImpl buf = new SessionInputBufferImpl(metrics, 2048);
buf.bind(is);
DefaultHttpRequestParser reqParser = new DefaultHttpRequestParser(buf);
HttpRequest req = reqParser.parse();
InputStream contentStream = null;
if (req instanceof HttpEntityEnclosingRequest) {
ContentLengthStrategy contentLengthStrategy = StrictContentLengthStrategy.INSTANCE;
long len = contentLengthStrategy.determineLength(req);
if (len == ContentLengthStrategy.CHUNKED) {
contentStream = new ChunkedInputStream(buf);
} else if (len == ContentLengthStrategy.IDENTITY) {
contentStream = new IdentityInputStream(buf);
} else {
contentStream = new ContentLengthInputStream(buf, len);
}
}
// Do something useful with the content stream (if non null)
Message parsers in HttpCore parse message heads only. However, one can proceed reading from the session input buffer and read message body content until the end of message (depending on the delineator used)
I am using commons-httpclient 3.1 to read a html page source. It is working fine with all except pages with content encoding as gzip. I am getting incomplete page source.
For this page firefox is showing content encoding as gzip.
Below are the details
Response header:
status code: HTTP/1.1 200 OK
Date = Wed, 20 Jul 2011 11:29:38 GMT
Content-Type = text/html; charset=UTF-8
X-Powered-By = JSF/1.2
Set-Cookie = JSESSIONID=Zqq2Tm8V74L1LJdBzB5gQzwcLQFx1khXNvcnZjNFsQtYw41J7JQH!750321853; path=/; HttpOnly
Transfer-Encoding = chunked
Content- length =-1
My code to read response :
HttpClient httpclient = new HttpClient();
httpclient.getParams().setParameter("http.connection.timeout",
new Integer(50000000));
httpclient.getParams().setParameter("http.socket.timeout",
new Integer(50000000));
// Create a method instance.
GetMethod method = new GetMethod(url);
// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
BufferedReader reader = null;
// Execute the method.
int statusCode = httpclient.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: "
+ method.getStatusLine());
strHtmlContent = null;
} else {
InputStream is = method.getResponseBodyAsStream();
reader = new BufferedReader(new InputStreamReader(is,"ISO8859_8"));
String line = null;
StringBuffer sbResponseBody = new StringBuffer();
while ((line = reader.readLine()) != null) {
sbResponseBody.append(line).append("\n");
}
strHtmlContent = sbResponseBody.toString();
Upgrade to httpclient 4.1. It should support compression seamlessly.
I just incurred in this issue, which I solved as follows:
URL url = new URL("http://www.megadevs.com");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
GZIPInputStream gzip = new GZIPInputStream(conn.getInputStream());
int value = -1;
String page = "";
while ((value = gzip.read()) != -1) {
char c = (char) value;
page += c;
}
gzip.close();
Hope this helps.
I use HttpURLConnection to do HTTP POST but I dont always get back the full response. I wanted to debug the problem, but when I step through each line it worked. I thought it must be a timing issue so I added Thread.sleep and it really made my code work, but this is only a temporary workaround. I wonder why is this happening and how to solve. Here is my code:
public static InputStream doPOST(String input, String inputMimeType, String url, Map<String, String> httpHeaders, String expectedMimeType) throws MalformedURLException, IOException {
URL u = new URL(url);
URLConnection c = u.openConnection();
InputStream in = null;
String mediaType = null;
if (c instanceof HttpURLConnection) {
//c.setConnectTimeout(1000000);
//c.setReadTimeout(1000000);
HttpURLConnection h = (HttpURLConnection)c;
h.setRequestMethod("POST");
//h.setChunkedStreamingMode(-1);
setAccept(h, expectedMimeType);
h.setRequestProperty("Content-Type", inputMimeType);
for(String key: httpHeaders.keySet()) {
h.setRequestProperty(key, httpHeaders.get(key));
if (logger.isDebugEnabled()) {
logger.debug("Request property key : " + key + " / value : " + httpHeaders.get(key));
}
}
h.setDoOutput(true);
h.connect();
OutputStream out = h.getOutputStream();
out.write(input.getBytes());
out.close();
mediaType = h.getContentType();
logger.debug(" ------------------ sleep ------------------ START");
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
logger.debug(" ------------------ sleep ------------------ END");
if (h.getResponseCode() < 400) {
in = h.getInputStream();
} else {
in = h.getErrorStream();
}
}
return in;
}
later I do the following to read the input stream
ByteArrayOutputStream bos = new ByteArrayOutputStream();
while (is.available() > 0) {
bos.write(is.read());
}
is.close();
//is.read(bytes);
if (logger.isDebugEnabled()) {
logger.debug(" Response lenght is : " + is.available());
//logger.debug("RAW response is " + new String(bytes));
logger.debug("RAW response is " + new String(bos.toByteArray()));
}
It genearates the following HTTP headers
POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39",oauth_nonce="YnDb5eepuLm%2Fbs",oauth_signature="dbN%2FWeWs2G00mk%2BX6uIi3thJxlM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276524919", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: keep-alive
Content-Length: 1107
In other posts it was suggested to turn off keep-alive by using the
http.keepAlive=false
system property, I tried that and the headers changed to
POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39", oauth_nonce="Eaiezrj6X4Ttt0", oauth_signature="ND9fAdZMqbYPR2j%2FXUCZmI90rSI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276526608", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: close
Content-Length: 1107
the Connection header is "close" but I still cannot read the whole response. Any idea what do I do wrong?
I think your problem is in this line:
while (is.available() > 0) {
According to the javadoc, available does not block and wait until all data is available, so you might get the first packet and then it will return false. The proper way to read from an InputStream is like this:
int len;
byte[] buffer = new byte[4096];
while (-1 != (len = in.read(buffer))) {
bos.write(buffer, 0, len);
}
Read will return -1 when there nothing left in the inputstream or the connection is closed, and it will block and wait for the network while doing so. Reading arrays is also much more performant than using single bytes.
Maybe I missed it, but what's the datatype of "input" in your code? Something that's strange about InputStreams in general is that the read( ... ) methods tend to block until data is available, then return only that data. You'll actually need to keep reading from your InputStream and appending to a ByteArrayInputStream or some other structure until you explicitly force a EOFException.
If you are reading the whole Message at once you can compare the isr.available() to the expected content lenght. This is how I did it:
public byte[] readData(HttpURLConnection conn)
throws IOException, InterruptedException {
String _connlen = conn.getHeaderField("Content-Length");
int connlen = Integer.parseInt(_connlen);
InputStream isr = null;
byte[] bytes = new byte[connlen];
try {
isr = conn.getInputStream();
//security count that it doesn't begin to hang
int maxcounter = 0;
//wait till all data is avalibal, max 5sec
while((isr.available() != connlen) && (maxcounter < 5000)){
Thread.sleep(1);
maxcounter++;
}
//Throw if not all data could be read
if(maxcounter >= 5000)
throw new IllegalAccessError();
//read the data
if(isr.read(bytes, 0, connlen) < 0)
throw new IllegalAccessError();
} finally {
if (isr != null)
isr.close();
}
return bytes;
}