I am using commons-httpclient 3.1 to read a html page source. It is working fine with all except pages with content encoding as gzip. I am getting incomplete page source.
For this page firefox is showing content encoding as gzip.
Below are the details
Response header:
status code: HTTP/1.1 200 OK
Date = Wed, 20 Jul 2011 11:29:38 GMT
Content-Type = text/html; charset=UTF-8
X-Powered-By = JSF/1.2
Set-Cookie = JSESSIONID=Zqq2Tm8V74L1LJdBzB5gQzwcLQFx1khXNvcnZjNFsQtYw41J7JQH!750321853; path=/; HttpOnly
Transfer-Encoding = chunked
Content- length =-1
My code to read response :
HttpClient httpclient = new HttpClient();
httpclient.getParams().setParameter("http.connection.timeout",
new Integer(50000000));
httpclient.getParams().setParameter("http.socket.timeout",
new Integer(50000000));
// Create a method instance.
GetMethod method = new GetMethod(url);
// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
BufferedReader reader = null;
// Execute the method.
int statusCode = httpclient.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: "
+ method.getStatusLine());
strHtmlContent = null;
} else {
InputStream is = method.getResponseBodyAsStream();
reader = new BufferedReader(new InputStreamReader(is,"ISO8859_8"));
String line = null;
StringBuffer sbResponseBody = new StringBuffer();
while ((line = reader.readLine()) != null) {
sbResponseBody.append(line).append("\n");
}
strHtmlContent = sbResponseBody.toString();
Upgrade to httpclient 4.1. It should support compression seamlessly.
I just incurred in this issue, which I solved as follows:
URL url = new URL("http://www.megadevs.com");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
GZIPInputStream gzip = new GZIPInputStream(conn.getInputStream());
int value = -1;
String page = "";
while ((value = gzip.read()) != -1) {
char c = (char) value;
page += c;
}
gzip.close();
Hope this helps.
Related
I am behind proxy server and I am using the following code to get data from url:
private static String getData(String address)throws Exception
{
System.setProperty("java.net.useSystemProxies","true");
Date d = new Date();
String finalAdress = address+"?x="+d.getTime();
URL url = new URL(finalAdress);
System.out.println(finalAdress);
InputStream html = null;
HttpsURLConnection con = (HttpsURLConnection)url.openConnection();
con.setUseCaches(false);
con.setDefaultUseCaches(false);
con.setRequestMethod("GET");
con.addRequestProperty("Cache-Control", "no-cache");
//con.addRequestProperty("Cache-Control", "max-age=0");
con.addRequestProperty("Expires", "0");
con.addRequestProperty("Pragma", "no-cache");
html = con.getInputStream();
//html = url.openStream();
int c = 0;
StringBuffer buffer = new StringBuffer("");
while(c != -1) {
c = html.read();
buffer.append((char)c);
}
return buffer.toString();
}
However when data changes on the server side - I sill for some time get the same old (cached) data as a response..
Tried to use:
-Cache-Control
-slightly modified urls address+"?x="+d.getTime();
but nothing seems to work.
Is there a way to force refresh as I would with a web browser (ctrl-F5) ?
I am trying to convert my POST request for multipart/form-data into java program and I am receiving java.io.IOException: Server returned HTTP response code: 400
// Connect to the web server endpoint
URL serverUrl =
new URL("http://testurl/start-date/2017-01-01/upload");
HttpURLConnection urlConnection = (HttpURLConnection) serverUrl.openConnection();
//String boundaryString = "----SomeRandomText";
String fileUrl = "C:/Users/V70070/Documents/ALL FILES/Rates/MAC2291_RATES.csv";
String encodedCredentials = "Basic dGVzdDpwYXNzd29yZA==";
File logFileToUpload = new File(fileUrl);
// Indicate that we want to write to the HTTP request body
urlConnection.setDoOutput(true);
urlConnection.setRequestProperty ("Authorization", encodedCredentials);
urlConnection.setRequestMethod("POST");
urlConnection.addRequestProperty("Content-Type", "multipart/form-data");
OutputStream outputStreamToRequestBody = urlConnection.getOutputStream();
BufferedWriter httpRequestBodyWriter =
new BufferedWriter(new OutputStreamWriter(outputStreamToRequestBody));
// Include value from the myFileDescription text area in the post data
httpRequestBodyWriter.write("Content-Disposition: form-data; name=\"fileData\"; Content-Type: text/plain; filename=\"MAC2291_RATES.csv\"");
// Include the section to describe the file
// httpRequestBodyWriter.write("Content-Disposition: form-data;"
// + "name=\"myFile\";"
// + "filename=\"" + logFileToUpload.getName() + "\""
// + "\nContent-Type: text/plain\n\n");
// httpRequestBodyWriter.flush();
// Write the actual file contents
FileInputStream inputStreamToLogFile = new FileInputStream(logFileToUpload);
int bytesRead;
byte[] dataBuffer = new byte[1024];
while ((bytesRead = inputStreamToLogFile.read(dataBuffer)) != -1) {
outputStreamToRequestBody.write(dataBuffer, 0, bytesRead);
}
outputStreamToRequestBody.flush();
// Mark the end of the multipart http request
//httpRequestBodyWriter.write("\n--" + boundaryString + "--\n");
httpRequestBodyWriter.flush();
// Read response from web server, which will trigger the multipart HTTP request to be sent.
BufferedReader httpResponseReader =
new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String lineRead;
while((lineRead = httpResponseReader.readLine()) != null) {
System.out.println(lineRead);
}
// Close the streams
outputStreamToRequestBody.close();
httpRequestBodyWriter.close();
}
I am getting the Exception while Reading response from web server
BufferedReader httpResponseReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
Any help highly appreciated. TIA
In the following code, EntityUtils.toString is going into IOException. When I paste 'EntityUtils.toString(entity)' on eclipse watch window, it showing me the value which is DISABLED
private String triggerRestApiCalls(String url){
HttpClient httpClient = new DefaultHttpClient();
HttpGet getRequest = new HttpGet(url);
getRequest.setHeader(
new BasicHeader("Accept", "application/json"));
try {
HttpResponse response = httpClient.execute(getRequest);
HttpEntity entity = response.getEntity();
String value = EntityUtils.toString(entity);
return value;
} catch (ClientProtocolException e) {
log.debug(e.getCause());
} catch (IOException e) {
log.debug(e.getCause());
}
log.debug("Status Unknown");
return "UNKNOWN";
}
The content value lenght is 8. The string expected is DISABLED, which is exactly of the length. The HTTP status is 200 (OK).
I used curl with same URL.
curl -i -H {Accept: application/json} http://127.0.0.1:9031/test/test.html?someDetails
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=D35C61744F4FB3A47B624FF3D0BEB026; Path=/mics/; Secure; HttpOnly
Content-Type: text/plain;charset=ISO-8859-1
Content-Length: 8
Date: Wed, 26 Nov 2014 13:23:30 GMT
DISABLED.
Any help is appreciated ! Is there any angle to encoding ?
FIRST EDIT
The Stack Trace mentions this.
java.io.IOException: Attempted read from closed stream.
The code that is executed by EntityUtils.toString()
public static String toString(
final HttpEntity entity, final Charset defaultCharset) throws IOException, ParseException {
if (entity == null) {
throw new IllegalArgumentException("HTTP entity may not be null");
}
InputStream instream = entity.getContent();
if (instream == null) {
return null;
}
try {
if (entity.getContentLength() > Integer.MAX_VALUE) {
throw new IllegalArgumentException("HTTP entity too large to be buffered in memory");
}
int i = (int)entity.getContentLength();
if (i < 0) {
i = 4096;
}
ContentType contentType = ContentType.getOrDefault(entity);
Charset charset = contentType.getCharset();
if (charset == null) {
charset = defaultCharset;
}
if (charset == null) {
charset = HTTP.DEF_CONTENT_CHARSET;
}
Reader reader = new InputStreamReader(instream, charset);
CharArrayBuffer buffer = new CharArrayBuffer(i);
char[] tmp = new char[1024];
int l;
while((l = reader.read(tmp)) != -1) {
buffer.append(tmp, 0, l);
}
return buffer.toString();
} finally {
instream.close();
}
}
I have stepped through this and there are not errors However, it closes the stream before it returns.
But the actual value that is returned is CharArrayBuffer which is not linked to the stream. The same code works in some other java file. Strange !! I am using spring.. is there a spring angle to this ?
The HttpEntity can only be read once and it seems something else is intercepting and reading the response so that when you attempt to apply EntityUtils.toString() you get this exception. I can't see why this would be happening, though you did mention there could be a Spring angle so there could be a Spring interceptor applied here.
You could try
String value = httpClient.execute(getRequest, new BasicResponseHandler());
Although from what I can see this should be fairly equivalent to the above code.
I'm trying to parse an HTTP request using the apache httpcore components and want to grab the body of the request. It looks like the default DefaultHttpRequestParser doesn't parse the body/entity from its input stream. Is there a class that will do this?
Unfortunately I can't use the entire stack and need to pull the request straight from this input stream.
My parsing code is below. Looking at some of the other answers it appears that the body of the request should be available as an entity. However, every time I try to get at the entity it is null.
Debugging I see that the buffer has read but not used the body and that DefaultHttpRequestParser seems to just read the header. Is there a parse I should be using to parse the entire input?
InputStream is = socket.getInputStream();
HttpTransportMetricsImpl metrics = new HttpTransportMetricsImpl();
SessionInputBufferImpl buf = new SessionInputBufferImpl(metrics, 2048);
buf.bind(is);
DefaultHttpRequestParser reqParser = new DefaultHttpRequestParser(buf);
HttpRequest req = reqParser.parse();
if (req instanceof HttpEntityEnclosingRequest) {
entity = ((HttpEntityEnclosingRequest)query).getEntity();
//... entity is always null
If I read the input stream I end up with:
POST / HTTP/1.1
User-Agent: curl/Q.XX.0 (linux-gnu) libcurl/Q.XX.0 OpenSSL/X.Y.Z zlib/A.B.C.D libidn/E.FF librtmp/G.H
Host: localhost:8088
Accept: */*
Content-Length: 333
Content-Type: multipart/form-data; boundary=----------------------------39203c7982df
------------------------------39203c7982df
Content-Disposition: form-data; name="fileupload"; filename="grun.sh"
Content-Type: application/octet-stream
#!/bin/bash -x
java -classpath lib/antlr-4.4-complete.jar:build/classes org.antlr.v4.runtime.misc.TestRig Typegroup "AHI" -tree
------------------------------39203c7982df--
[Update] Oleg has a good answer, but can I associate the body with the request or do
I now need to pass two things around, the body and the stream? I'll be looking into
I got the following to work, but it's deprecated in the latest release.
...
HttpEntityEnclosingRequest ereq = (HttpEntityEnclosingRequest) req;
#SuppressWarnings("deprecation")
EntityDeserializer ed =
new EntityDeserializer(new LaxContentLengthStrategy());
#SuppressWarnings("deprecation")//ack!
HttpEntity ent = ed.deserialize(buf, req);
ereq.setEntity(ent);
return ereq;
Combining Oleg's solution with the above I ended up with:
HttpEntityEnclosingRequest ereq = (HttpEntityEnclosingRequest) req;
ContentLengthStrategy contentLengthStrategy =
StrictContentLengthStrategy.INSTANCE;
long len = contentLengthStrategy.determineLength(req);
InputStream contentStream = null;
if (len == ContentLengthStrategy.CHUNKED) {
contentStream = new ChunkedInputStream(buf);
} else if (len == ContentLengthStrategy.IDENTITY) {
contentStream = new IdentityInputStream(buf);
} else {
contentStream = new ContentLengthInputStream(buf, len);
}
BasicHttpEntity ent = new BasicHttpEntity();
ent.setContent(contentStream);
ereq.setEntity(ent);
return ereq;
InputStream is = socket.getInputStream();
HttpTransportMetricsImpl metrics = new HttpTransportMetricsImpl();
SessionInputBufferImpl buf = new SessionInputBufferImpl(metrics, 2048);
buf.bind(is);
DefaultHttpRequestParser reqParser = new DefaultHttpRequestParser(buf);
HttpRequest req = reqParser.parse();
InputStream contentStream = null;
if (req instanceof HttpEntityEnclosingRequest) {
ContentLengthStrategy contentLengthStrategy = StrictContentLengthStrategy.INSTANCE;
long len = contentLengthStrategy.determineLength(req);
if (len == ContentLengthStrategy.CHUNKED) {
contentStream = new ChunkedInputStream(buf);
} else if (len == ContentLengthStrategy.IDENTITY) {
contentStream = new IdentityInputStream(buf);
} else {
contentStream = new ContentLengthInputStream(buf, len);
}
}
// Do something useful with the content stream (if non null)
Message parsers in HttpCore parse message heads only. However, one can proceed reading from the session input buffer and read message body content until the end of message (depending on the delineator used)
I have an app running on J2ME that needs to access the Instagram API. Specifically, I need to post a comment. This is a Post method. So I try to add the "text" parameter to the body of the request by using HttpConnection.setRequestProperty(), but this doesn't seem to work, as Instagram doesn't recognize that the parameter is present. I think this method is failing to write the parameter to the body of the Http request. Any idea how I can make this work in j2me?
Here's my code:
InputStream is = null;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] response = null;
HttpConnection connection = null;
String url = "";
try {
url = "https://api.instagram.com/v1/media/" + mediaId + "/comments?access_token="+InstagramAPIUtil.accessTokenTest;// POST
url = Util.urlEncodeSpaces(url);
System.out.println(url);
connection = (HttpConnection)Connector.open(url,Connector.READ_WRITE);
connection.setRequestMethod(HttpConnection.POST);
connection.setRequestProperty("text", comment);
if (connection.getResponseCode() == HttpConnection.HTTP_OK) {
is = connection.openInputStream();
if (is != null) {
int ch = -1;
while ((ch = is.read()) != -1) {
bos.write(ch);
}
response = bos.toByteArray();
}
System.out.println("response: "+new String(response));
System.out.println("request: "+connection.getRequestProperty("text"));
return true;
}
And here's what I'm getting back from Instagram:
{"meta":{"error_type":"APIInvalidParametersError","code":400,"error_message":"Missing 'text'"}}
I have not much experience in the HTTP area, but I think you need to write to the output stream that you can get from connection. I don't see where in your code you send actual data.
HttpURLConnection c = (HttpURLConnection) new URL("").openConnection();
OutputStream out = c.getOutputStream();
InputStream in = c.getInputStream();