I have a bolt that is making an API call (HTTP Get) for every tuple.
to avoid the need to wait for the response, I was looking to use the apache HttpAsyncClient.
after instantiating the client in the bolt's prepare method, the execute method constructs the URL from the tuple and calls sendAsyncGetRequest(url):
private void sendAsyncGetRequest(String url){
httpclient.execute(new HttpGet(url), new FutureCallback<HttpResponse>() {
#Override
public void completed(HttpResponse response) {
LOG.info("Response Code : " + response.getStatusLine());
LOG.debug(response.toString());
}
#Override
public void failed(Exception ex) {
LOG.warn("Async http request failed!", ex);
}
#Override
public void cancelled() {
LOG.warn("Async http request canceled!");
}
});
}
the topology deploys but the Storm UI shows an error:
java.lang.RuntimeException: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:12
I got this to work with no issues.
the key things to note are:
declare the client on the bolt class scope
public class MyRichBolt extends BaseRichBolt {
private CloseableHttpAsyncClient httpclient;
Instantiate and stat the client in the bolt's prepare method
#Override
public final void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
try {
// start the http client
httpclient = HttpAsyncClients.createDefault();
httpclient.start();
// other initialization code ...
} catch (Throwable exception) {
// handle errors
}
}
make the calls in the bolt's execute method
#Override
public final void execute(Tuple tuple) {
// format the request url
String url = ...
sendAsyncGetRequest(url);
}
private void sendAsyncGetRequest(String url){
logger.debug("Async call to URL...");
HttpGet request = new HttpGet(url);
HttpAsyncRequestProducer producer = HttpAsyncMethods.create(request);
AsyncCharConsumer<HttpResponse> consumer = new AsyncCharConsumer<HttpResponse>() {
HttpResponse response;
#Override
protected void onResponseReceived(final HttpResponse response) {
this.response = response;
}
#Override
protected void onCharReceived(final CharBuffer buf, final IOControl ioctrl) throws IOException {
// Do something useful
}
#Override
protected void releaseResources() {
}
#Override
protected HttpResponse buildResult(final HttpContext context) {
return this.response;
}
};
httpclient.execute(producer, consumer, new FutureCallback<HttpResponse>() {
#Override
public void completed(HttpResponse response) {
// do something useful with the response
logger.debug(response.toString());
}
#Override
public void failed(Exception ex) {
logger.warn("!!! Async http request failed!", ex);
}
#Override
public void cancelled() {
logger.warn("Async http request canceled!");
}
});
}
Are you shutting down the client (client.close();) in your main flow before the callback can execute?
The error is saying that the IO path has already been closed. In general, instances of async clients should be re-used for repeated requests and destroyed only when "ALL" requests have been made, e.g. at application shutdown.
Related
I have created a spring boot Messaging endpoint and need to create an android chat app and am wondering how I can manage to call those endpoints using okttp Websocket client which does not seem to have a way to add api endpoints like this javascript code.
And here is my spring boot endpoints
#Configuration
#EnableWebSocketMessageBroker
public class WebMessageConfig implements WebSocketMessageBrokerConfigurer {
#Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableSimpleBroker( "/user");
config.setApplicationDestinationPrefixes("/app");
config.setUserDestinationPrefix("/user");
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry
.addEndpoint("/ws")
.withSockJS()
.setAllowedOrigins("*");
}
}
And here is my OkHttp client code
public class StompWs {
private String SERVER_PATH="ws://mydomain.com:8443/MyContex/ws";
public static void main(String[] args) {
try {
new StompWs().run();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private WebSocket webSocket;
public void run() throws Exception {
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder().url(SERVER_PATH).build();
webSocket = client.newWebSocket(request, new SocketListener());
}
private String getData()
{
MessageModel message=new MessageModel();
message.setMessage("Hello");
message.setRecipientId("1");
message.setSenderId("2");
return new Gson().toJson(message);
}
private class SocketListener extends WebSocketListener {
#Override
public void onOpen(WebSocket webSocket, Response response) {
super.onOpen(webSocket, response);
try {
webSocket.send(getData());
/**I need equivalent of this
stompClient.subscribe(
"/user/1/queue/messages",// I need java code to do this
onMessageReceived
*/
}
catch(Exception e)
{
e.printStackTrace();
}
System.out.println("succesfully connected:"+response.toString());//this message execute well
}
#Override
public void onMessage(WebSocket webSocket, String text) {
super.onMessage(webSocket, text);
System.out.println("on message:"+text);
}
#Override
public void onFailure(WebSocket webSocket, Throwable t,
Response response) {
// TODO Auto-generated method stub
super.onFailure(webSocket, t, response);
System.out.println("on message:"+t.toString());
}
}
}
This question was asked for Apache Commons HttpClient, however I'm using the async client HttpAsyncClient.
Content decompression (gzip) does not work out of the box.
I tried to configure it with:
httpClientAsync = HttpAsyncClients.custom()
.setMaxConnPerRoute(100)
.setMaxConnTotal(200)
// Enable response content encoding (gzip)
//
// NOTE: Does not work for some reason
.addInterceptorLast(ResponseContentEncoding())
Which I copied from HttpClientBuilder, but it doesn't work.
Any ideas?
The use of addInterceptorLast and addInterceptorFirst has no effect.
asyncHttpClient.execute() will create a BasicAsyncResponseConsumer by default.
This BasicAsyncResponseConsumer will copy the original ContentDecoder into the buffer, resulting in DecompressingEntity.getContent() is never called.
org.apache.http.impl.nio.client.CloseableHttpAsyncClient#execute()
public Future<HttpResponse> execute(
final HttpHost target, final HttpRequest request, final HttpContext context,
final FutureCallback<HttpResponse> callback) {
return execute(
HttpAsyncMethods.create(target, request),
HttpAsyncMethods.createConsumer(), // BasicAsyncResponseConsumer
context, callback);
}
org.apache.http.nio.protocol.BasicAsyncResponseConsumer#onContentReceived
protected void onContentReceived(
final ContentDecoder decoder, final IOControl ioControl) throws IOException {
Asserts.notNull(this.buf, "Content buffer");
this.buf.consumeContent(decoder);
}
My solution is to manually call ResponseContentEncoding.process(resp, context) in the callback to reset the HttpEntity.
private static final ResponseContentEncoding responseContentEncoding = new ResponseContentEncoding();
HttpClientContext hcc = HttpClientContext.create();
asyncHttpClient.execute(bidreq, hcc, new FutureCallback<HttpResponse>() {
#Override
public void completed(HttpResponse result) {
HttpEntity entity = null;
String content = null;
try {
responseContentEncoding.process(result, hcc);
entity = result.getEntity();
if (entity != null) {
content = EntityUtils.toString(entity, UTF_8);
log.info(content);
}
} catch (Exception e) {
log.error("error", e);
} finally {
EntityUtils.consumeQuietly(entity);
}
}
#Override
public void failed(Exception ex) {
log.error("failed", ex);
}
#Override
public void cancelled() { }
});
I want to close the channel when it hasn't received any data after certain seconds. I tried IdleHandler, but it isn't working. My main handler is clientHandler which extends SimpleChannelInboundHandler. This sends data in string and receives data in String format. Sometimes, I don't get the data during that time I want my channel to close after certain timeout, but currently it is waiting for the data from the server.
One more observation, When I check in the packet sender to verify for the same request. I get empty response from the server but this response is not received by my ClientHandler.
Following is the code.
clientBootstrap.handler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch){
ch.pipeline()
.addLast(new IdleStateHandler(5, 5, 10))
.addLast(new MyHandler())
.addLast(new ClientHandler(cardIssueRequest,promise));
}
});
MyHandler:
public class MyHandler extends ChannelDuplexHandler {
#Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) {
if (evt instanceof IdleStateEvent) {
IdleStateEvent e = (IdleStateEvent) evt;
if (e.state() == IdleState.READER_IDLE) {
ctx.close();
} else if (e.state() == IdleState.WRITER_IDLE) {
ctx.close();
}
}
}
}
ClientHandler:
public class ClientHandler extends SimpleChannelInboundHandler {
RequestModel request;
private final Promise<String> promise;
public ClientHandler(RequestModel request, Promise<String> promise) {
this.request = request;
this.promise = promise;
}
#Override
protected void channelRead0(ChannelHandlerContext channelHandlerContext, Object o) {
String response = ((ByteBuf) o).toString(CharsetUtil.UTF_8);
log.info("Client received: " + response);
promise.trySuccess(response);
}
#Override
public void channelActive(ChannelHandlerContext channelHandlerContext) {
log.info("Client sent: " + request);
channelHandlerContext.writeAndFlush(Unpooled.copiedBuffer((request.toString()), CharsetUtil.UTF_8));
}
#Override
public void exceptionCaught(ChannelHandlerContext channelHandlerContext, Throwable cause) {
cause.printStackTrace();
channelHandlerContext.close();
promise.setFailure(cause);
}
}
After taking the thread dump, I found the issue was that my program was waiting in the promise statement. So, after setting timeout for the promise, my issue got solved.
promise.get(60, TimeUnit.SECONDS)
I am making a Curl post curl -X POST -d "dsds" 10.0.0.211:5201 to my Netty socket server but in my ChannelRead when I try to cast Object msg into FullHttpRequest It throws following exception.
java.lang.ClassCastException: io.netty.buffer.SimpleLeakAwareByteBuf cannot be cast to io.netty.handler.codec.http.FullHttpRequest
at edu.clemson.openflow.sos.host.netty.HostPacketHandler.channelRead(HostPacketHandler.java:42)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:326)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1320)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:905)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:563)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
at java.lang.Thread.run(Thread.java:748)
Following is my Socket Handler class
#ChannelHandler.Sharable
public class HostPacketHandler extends ChannelInboundHandlerAdapter {
private static final Logger log = LoggerFactory.getLogger(HostPacketHandler.class);
private RequestParser request;
public HostPacketHandler(RequestParser request) {
this.request = request;
log.info("Expecting Host at IP {} Port {}",
request.getClientIP(), request.getClientPort());
}
public void setRequestObject(RequestParser requestObject) {
this.request = requestObject;
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
// Discard the received data silently.
InetSocketAddress socketAddress = (InetSocketAddress) ctx.channel().remoteAddress();
log.info("Got Message from {} at Port {}",
socketAddress.getHostName(),
socketAddress.getPort());
//FullHttpRequest request = (FullHttpRequest) msg;
log.info(msg.getClass().getSimpleName());
//((ByteBuf) msg).release();
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
// Close the connection when an exception is raised.
cause.printStackTrace();
ctx.close();
}
}
Pipeline:
public class NettyHostSocketServer implements IClientSocketServer {
protected static boolean isClientHandlerRunning = false;
private static final Logger log = LoggerFactory.getLogger(SocketManager.class);
private static final int CLIENT_DATA_PORT = 9877;
private static final int MAX_CLIENTS = 5;
private HostPacketHandler hostPacketHandler;
public NettyHostSocketServer(RequestParser request) {
hostPacketHandler = new HostPacketHandler(request);
}
private boolean startSocket(int port) {
NioEventLoopGroup group = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(group)
.channel(NioServerSocketChannel.class)
.localAddress(new InetSocketAddress(port))
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch)
throws Exception {
ch.pipeline().addLast(
hostPacketHandler);
}
});
ChannelFuture f = b.bind().sync();
log.info("Started host-side socket server at Port {}",CLIENT_DATA_PORT);
return true;
// Need to do socket closing handling. close all the remaining open sockets
//System.out.println(EchoServer.class.getName() + " started and listen on " + f.channel().localAddress());
//f.channel().closeFuture().sync();
} catch (InterruptedException e) {
log.error("Error starting host-side socket");
e.printStackTrace();
return false;
} finally {
//group.shutdownGracefully().sync();
}
}
#Override
public boolean start() {
if (!isClientHandlerRunning) {
isClientHandlerRunning = true;
return startSocket(CLIENT_DATA_PORT);
}
return true;
}
#Override
public int getActiveConnections() {
return 0;
}
}
I also used wireshark to check If I am getting valid packets or not. Below is the screenshot of Wireshark dump.
Your problem is that you never decode the ByteBuf into an actual HttpRequest object which is why you get an error. You can't cast a ByteBuf to a FullHttpRequest object.
You should do something like this:
#Override
public void initChannel(Channel channel) throws Exception {
channel.pipeline().addLast(new HttpRequestDecoder()) // Decodes the ByteBuf into a HttpMessage and HttpContent (1)
.addLast(new HttpObjectAggregator(1048576)) // Aggregates the HttpMessage with its following HttpContent into a FullHttpRequest
.addLast(hostPacketHandler);
}
(1) If you also want to send HttpResponse use this handler HttpServerCodec which adds the HttpRequestDecoder and HttpResponseEncoder.
I'm using okhttp3 with asynchronous callbacks to get JSONArrays/JSONObjects from the server and then parsing them and creating the particular object which is passed to the callback function.
Most of the callback code is the same for every method, but there are some lines of code that differ.
Is there a pattern that I can use to reduce the lines of code so that I don't have to write the same code over and over again for the different objects?
I marked the lines of code that differ for every method.
The problem I have is calling the particular JSON parsing function without using switch/case and varying the callback object.
//-----------------------differs-------------------------
public void getUser(final HTTPResponseCallback<User> callback)
{
//-----------------------differs-------------------------
final String url = domain + USERS;
//-------------------------------------------------------
okHttpClient.newCall(buildRequest(url)).enqueue(new Callback()
{
Handler handler = new Handler(Looper.getMainLooper());
#Override
public void onFailure(Call call, IOException e)
{
handler.post(new Runnable()
{
#Override
public void run()
{
callback.onFailure();
}
});
}
#Override
public void onResponse(Call call, final Response response) throws IOException
{
if (response.isSuccessful())
{
try
{
String responseBody = response.body().string();
//-----------------------differs-------------------------
JSONObject jsonResponse = new JSONObject(responseBody);
final User user = JsonParser.parseUser(jsonResponse
//------------------------------------------------------
handler.post(new Runnable()
{
#Override
public void run()
{
//---------------------------------------last parameter differs----------------------------------------------
callback.onSuccess(new HTTPTransaction(response.code(), response.message(), response.header("ETag")), user);
//-----------------------------------------------------------------------------------------------------------
}
});
}
catch (JSONException e)
{
...
}
}
else
...
}
}
}
1) Make in parameterized with <T> as type can differs:
public class CallBackWrapper<T> {
...
public void getUser(final HTTPResponseCallback<T> callback) { ...
2) Introduce callback object for unique parts which will return instance of type T:
interface Worker {
T run(String responseBody);
}
public <T> void getUser(final HTTPResponseCallback<T> callback, Worker worker) { ...
3) Invoke needed worker:
String responseBody = response.body().string();
//-----------------------differs-------------------------
final T obj = worker.run(responseBody);
//------------------------------------------------------
handler.post(new Runnable()
{
#Override
public void run()
{
//---------------------------------------last parameter differs----------------------------------------------
callback.onSuccess(new HTTPTransaction(response.code(), response.message(), response.header("ETag")), obj);
//-----------------------------------------------------------------------------------------------------------
}
});