I'm new to Java development so please bear with me. Also, I hope I'm not the champion of tl;dr :).
I'm using HttpClient to make requests over Http (duh!) and I'd gotten it to work for a simple servlet that receives an URL as a query string parameter. I realized that my code could use some refactoring, so I decided to make my own HttpResponseHandler, to clean up the code, make it reusable and improve exception handling.
I currently have something like this:
public class HttpResponseHandler implements ResponseHandler<InputStream>{
public InputStream handleResponse(HttpResponse response)
throws ClientProtocolException, IOException {
int statusCode = response.getStatusLine().getStatusCode();
InputStream in = null;
if (statusCode != HttpStatus.SC_OK) {
throw new HttpResponseException(statusCode, null);
} else {
HttpEntity entity = response.getEntity();
if (entity != null) {
in = entity.getContent();
// This works
// for (int i;(i = in.read()) >= 0;) System.out.print((char)i);
}
}
return in;
}
}
And in the method where I make the actual request:
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(target);
ResponseHandler<InputStream> httpResponseHandler = new HttpResponseHandler();
try {
InputStream in = httpclient.execute(httpget, httpResponseHandler);
// This doesn't work
// for (int i;(i = in.read()) >= 0;) System.out.print((char)i);
return in;
} catch (HttpResponseException e) {
throw new HttpResponseException(e.getStatusCode(), null);
}
The problem is that the input stream returned from the handler is closed. I don't have any idea why, but I've checked it with the prints in my code (and no, I haven't used them both at the same time :). While the first print works, the other one gives a closed stream error.
I need InputStreams, because all my other methods expect an InputStream and not a String. Also, I want to be able to retrieve images (or maybe other types of files), not just text files.
I can work around this pretty easily by giving up on the response handler (I have a working implementation that doesn't use it), but I'm pretty curious about the following:
Why does it do what it does?
How do I open the stream, if something closes it?
What's the right way to do this, anyway :)?
I've checked the docs and I couldn't find anything useful regarding this issue. To save you a bit of Googling, here's the Javadoc and here's the HttpClient tutorial (Section 1.1.8 - Response handlers).
Thanks,
Alex
It closes the stream because ResponseHandler must handle the whole response. Even if you get an open stream, it should be at the end of stream.
The stream is closed by BasicHttpEntity's consumeContent() call to ensure you don't read from the stream again.
In your case, you don't really need ResponseHandler.
The automatic resource management which is called closes the stream for you to make sure all resources are freed and ready for the next task.
If you want streams then you best bet is to copy it to a ByteArray and return a ByteArrayInputStream if the content is relatively modest.
If the content is not modest, then you'll have to do the resource management your self and not the the ResponseHandler.
Related
At work we use Netflix's Feign Client to help with requests between services. However, I'm confused about its apparent lack of ability to stream data, especially given Netflix's well known business model of streaming video. I clearly am missing something here.
To explain, say Service A asks the Feign Client of Service B for a stream of data and Service B sends the stream in the response. At this point, the execute() method in the Feign Client gets called:
#Override public Response execute(Request request, Options options) throws IOException {
HttpURLConnection connection = convertAndSend(request, options);
return convertResponse(connection);
}
HttpURLConnection convertAndSend(Request request, Options options) throws IOException {
final HttpURLConnection connection = (HttpURLConnection) new URL(request.url()).openConnection();
/** SNIP **/
if (request.body() != null) {
if (contentLength != null) {
connection.setFixedLengthStreamingMode(contentLength);
} else {
connection.setChunkedStreamingMode(8196);
}
connection.setDoOutput(true);
OutputStream out = connection.getOutputStream();
if (gzipEncodedRequest) {
out = new GZIPOutputStream(out);
}
try {
out.write(request.body()); // PROBLEM
} finally {
try {
out.close();
} catch (IOException suppressed) {
}
}
}
return connection;
}
The line labelled PROBLEM is what confuses me.
The request object doesn't even have any sort of stream to read, just a byte[] body.
On the outgoing end, the entire body is written into the OutputStream at once. Shouldn't it chunk the data instead?
For example
// pseudocode
try {
location = 0
bufferSize = 2048
buffer = request.body().read(location, bufferSize)
while(out.readyToRead() && buffer.length > 0) {
out.write(buffer)
location += bufferSize
buffer = request.body().read(location, bufferSize)
}
}
If the request had a stream instead of just byte[] body, you could improve that even further to send data as it becomes available.
I'm very new to this area of service architecture. What am I missing?
Feign was designed for control plane apis, which often don't benefit by streaming upwards. Streaming downwards is supported, though.
I've no concern with being more efficient with regards to how buffering works (ex. alternative to byte array). Just bear in mind that most of feign's design revolves around templating forms (json or xml) and reusing these as much as possible (ex. on retransmit, buffered + fixed length is easy and predictable).
I think I'd be most happy about a "streaming" design if it were coupled to the http client. IOTW, a subtype that addresses streaming in a way that makes sense in the transport. For example, InputStream for regular java, OkIo buffer for OkHttp, Netty Buffer for Netty, etc.
Spencer opened this for the investigation https://github.com/Netflix/feign/issues/220
I am new to netty and I followed this example to write a static file server using netty. But whenever the server serves a large js file. It runs into ClosedChannelException.
The following is my code where I write chunkedFile as http response.
When a large js file is being served I get closedChannelException and the raf file is also closed.
Could you help me figure out what I have done wrong here? Also, is there a simple tutorial where I get understand the basic flow of control in netty?
// Write the content.
ChannelFuture writeFuture = null;
try
{
long fileLength = raf.length();
HttpResponse response = new DefaultHttpResponse(
HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
response.setHeader(HttpHeaders.Names.CONTENT_LENGTH, fileLength);
Channel c = ctx.getChannel();
// Write the initial line and the header.
c.write(response);
writeFuture = c.write(new ChunkedFile(raf, 0, fileLength, 8192));
}
finally
{
raf.close();
if (writeFuture != null)
writeFuture.addListener(ChannelFutureListener.CLOSE);
}
}
<
Calling raf.close() in the finally block is wrong as it may not have it written yet. In fact netty will take care to close it after the write is complete.
I am using Apache HTTPClient 4 to connect to twitter's streaming api with default level access. It works perfectly well in the beginning but after a few minutes of retrieving data it bails out with this error:
2012-03-28 16:17:00,040 DEBUG org.apache.http.impl.conn.SingleClientConnManager: Get connection for route HttpRoute[{tls}->http://myproxy:80->https://stream.twitter.com:443]
2012-03-28 16:17:00,040 WARN com.cloudera.flume.core.connector.DirectDriver: Exception in source: TestTwitterSource
java.lang.IllegalStateException: Invalid use of SingleClientConnManager: connection still allocated.
at org.apache.http.impl.conn.SingleClientConnManager.getConnection(SingleClientConnManager.java:216)
Make sure to release the connection before allocating another one.
at org.apache.http.impl.conn.SingleClientConnManager$1.getConnection(SingleClientConnManager.java:190)
I understand why I am facing this issue. I am trying to use this HttpClient in a flume cluster as a flume source. The code looks like this:
public Event next() throws IOException, InterruptedException {
try {
HttpHost target = new HttpHost("stream.twitter.com", 443, "https");
new BasicHttpContext();
HttpPost httpPost = new HttpPost("/1/statuses/filter.json");
StringEntity postEntity = new StringEntity("track=birthday",
"UTF-8");
postEntity.setContentType("application/x-www-form-urlencoded");
httpPost.setEntity(postEntity);
HttpResponse response = httpClient.execute(target, httpPost,
new BasicHttpContext());
BufferedReader reader = new BufferedReader(new InputStreamReader(
response.getEntity().getContent()));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>30000) break;
}
return new EventImpl(buffer.toString().getBytes());
} catch (IOException ie) {
throw ie;
}
}
I am trying to buffer 30,000 characters in the response stream to a StringBuffer and then return this as the data received. I am obviously not closing the connection - but I do not want to close it just yet I guess. Twitter's dev guide talks about this here It reads:
Some HTTP client libraries only return the response body after the
connection has been closed by the server. These clients will not work
for accessing the Streaming API. You must use an HTTP client that will
return response data incrementally. Most robust HTTP client libraries
will provide this functionality. The Apache HttpClient will handle
this use case, for example.
It clearly tells you that HttpClient will return response data incrementally. I've gone through the examples and tutorials, but I haven't found anything that comes close to doing this. If you guys have used a httpclient (if not apache) and read the streaming api of twitter incrementally, please let me know how you achieved this feat. Those who haven't, please feel free to contribute to answers. TIA.
UPDATE
I tried doing this: 1) I moved obtaining stream handle to the open method of the flume source. 2) Using a simple inpustream and reading data into a bytebuffer. So here is what the method body looks like now:
byte[] buffer = new byte[30000];
while (true) {
int count = instream.read(buffer);
if (count == -1)
continue;
else
break;
}
return new EventImpl(buffer);
This works to an extent - I get tweets, they are nicely being written to a destination. The problem is with the instream.read(buffer) return value. Even when there is no data on the stream, and the buffer has default \u0000 bytes and 30,000 of them, so this value is getting written to the destination. So the destination file looks like this.. " tweets..tweets..tweeets.. \u0000\u0000\u0000\u0000\u0000\u0000\u0000...tweets..tweets... ". I understand the count won't return a -1 coz this is a never ending stream, so how do I figure out if the buffer has new content from the read command?
The problem is that your code is leaking connections. Please make sure that no matter what you either close the content stream or abort the request.
InputStream instream = response.getEntity().getContent();
try {
BufferedReader reader = new BufferedReader(
new InputStreamReader(instream));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if (buffer.length()>30000) {
httpPost.abort();
// connection will not be re-used
break;
}
}
return new EventImpl(buffer.toString().getBytes());
} finally {
// if request is not aborted the connection can be re-used
try {
instream.close();
} catch (IOException ex) {
// log or ignore
}
}
It turns out that it was a flume issue. Flume is optimized to transfer events of size 32kb. Anything beyond 32kb, Flume bails out. (The workaround is to tune event size to be greater than 32KB). So, I've changed my code to buffer 20,000 characters at least. It kind of works, but it is not fool proof. This can still fail if the buffer length exceeds 32kb, however, it hasn't failed so far in an hour of testing - I believe it has to do with the fact that Twitter doesn't send a lot of data on its public stream.
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>20000) break;
}
What would be the proper way to simply see if a connection to a website/server can be made? I want this for an application I am coding that will just alert me if my website goes offline.
Thanks!
You can use an HttpURLConnection to send a request and check the response body for text that is unique to that page (rather than just checking to see if there's a response at all, just in case an error or maintenance page or something is being served).
Apache Commons has a library that removes a lot of the boiler plate of making Http requests in Java.
I've never done anything like this specifically on Android, but I'd be surprised if it's any different.
Here's a quick example:
URL url = new URL(URL_TO_APPLICATION);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
InputStream stream = connection.getInputStream();
Scanner scanner = new Scanner(stream); // You can read the stream however you want. Scanner was just an easy example
boolean found = false;
while(scanner.hasNext()) {
String next = scanner.next();
if(TOKEN.equals(next)) {
found = true;
break;
}
}
if(found) {
doSomethingAwesome();
} else {
throw aFit();
}
You want to also set the connection timeout using setConnectTimeout(int timeout) and setReadTimeout(int timeout). Otherwise the code might hang for a long time waiting for a non-responding server to reply.
Do anyone know how to copy data in zip file, jar file , binary file and others in REST web service using java? I write a web service method to copy file using FileInputStream , but it can only copy file type.
thanks
I'd recommend using apache httpclient for this. Your code might look something like (note, make sure you're using version 4.x or higher):
HttpClient client = new DefaultHttpClient();
HttpRequestBase httpMethod = httpMethod = new HttpGet(myUrlString);
httpMethod.setHeader("Accept", "application/zip");
HttpResponse response = httpClient.execute(httpMethod);
int statusCode = response.getStatusLine().getStatusCode();
if(statusCode != 200) {
throw new Exception("Bad return status code of: "+statusCode);
}
HttpEntity entity = response.getEntity();
if( entity != null) {
FileOutputStream fos = new FileOutputStream("myFile.zip");
int nextByte=0;
InputStream cis = entity.getContent();
try {
while( (nextByte = cis.read()) >= 0) fos.write(nextByte);
} finally {
fos.close();
cis.close();
}
}
I haven't compiled this, but you could probably get it going without too much issue (feel free to edit my comment and correct the code if you try to compile this and there are errors). Also note, this code should generically work for downloading anything from a web request (after changing the "Accept" header).