I'm using a Spring WebFlux WebSocketClient to subscribe to and handle messages from a remote web socket. During processing the Flux of messages from the remote socket will sometimes unexpectedly complete (or terminate on error) causing the web socket client's onComplete (or onError) callback to execute. When this occurs, my onComplete and onError callbacks publish an event. An event listener responds by calling the function that creates another web socket client which connects to the same external web socket and the socket processing starts over again.
My problem is that I cannot figure out how to free the WebSocketClient resources after a client completes processing. This causes unused threads to accumulate in the JVM. In particular the threads on which the first WebSocketClient were running (WebSocketClient-SecureIO-1, WebSocketClient-SecureIO-2 and parallel-1) remain in a waiting state and new threads are started for the new 'WebSocketClient'. I thought calling close() on WebSocketSession would solve the problem, but it does not.
The pattern of my implementation is:
public void startProcessing() {
WebSocketClient client = new StandardWebSocketClient();
Mono<String> subscribeMsg = Mono.just("...");
client
.execute(uri, webSocketSession ->
webSocketSession
.send(subscribeMsg.map(webSocketSession::textMessage))
.thenMany(webSocketSession.receive())
.map(webSocketMessage -> ...)
.buffer(Duration.ofSeconds(bufferDuration))
.doOnNext(handler)
.doOnComplete(() -> webSocketSession.close())
.then())
.subscribe(
aVoid -> LOGGER.info("subscription started"),
throwable -> {... publish restart event ...},
() -> {... publish restart event ...});
}
public void restartEventListener() {
startProcessing();
}
Any suggestions on how I can prevent unused WebSocketClient threads from accumulating in the JVM?
A few ideas:
A WebSocketClient is pooling resources, so you should reuse the same client for many requests.
You should avoid doing processing inside doOn* operators. Those are side-effects operators and are executed synchronously on the current Scheduler. Fore more efficiency, you should use other operators. You could map the websocket message to a Flux<DataBuffer> and then use DataBufferUtils::write to write those to a file and still leverage the same reactive pipeline instead of using the side-effects operators.
Closing the websocket session in one of those is not a bad idea, although I'd use doOnTerminate which is triggered for both success and error scenarios.
Also I don't understand the goal of publishing events to restart the processing phase. Using the retry and repeat operators and the same client should work just fine and be more efficient.
Related
I am reading the documentation about Channel.basicCancel operation in rabbitmq https://www.rabbitmq.com/consumer-cancel.html . The docs says that one of possible cancellation case is when consumer sends cancel signal on the same channel on which it is listening.
Is this the only possibility? Can you cancel remote consumer running on different channel/connection/process?
I am trying to send the cancel request from another another process. When I do it ends with an exception java.io.IOException: Unknown consumerTag just like such operation was restricted to cancelling local consumers (on own channel or connection).
UPDATE:
I noticed that this "Unknown consumerTag" exception is a result of initial validation inside com.rabbitmq.client.impl.ChannelN.basicCancel(String):
Consumer originalConsumer = (Consumer)this._consumers.get(consumerTag);
if (originalConsumer == null) {
throw new IOException("Unknown consumerTag");
}
...
But still there might be some rpc call which does the trick...
The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
The documentation is correct, you must cancel a consumer from its own channel/connection.
Other options include making your consumers aware of "cancellation messages" that will cause them to stop themselves, or using the API to close an entire connection, which will close all channels associated with it.
I have a simple MQTT listener that subscribes to a topic and call back
MqttClient client = new MqttClient(mqttHost, MqttClient.generateClientId());
client.connect();
client.subscribe("test", QUALITY_OF_SERVICE_2, new IMqttMessageListener() {
public void messageArrived(final String s, final MqttMessage mqttMessage) {
System.out.println("Received"+mqttMessage.toString());
// Code that blocks the thread
lock.lock();
//do something
lock.unlock();
});
Lets say i am publishing 1000 messages to the topic test but running the above listener on tomcat would display < 1000 console outputs showing that the receiver thread is not getting all the sent messages.
Without the lock() code, the listener works as expected and receives all messages.
You should not be doing long running/blocking tasks in the messageArrived handler, as this is called on the main network loop of the client.
If you have long running/blocking tasks to do with a message you should create a local queue and process the messages from that queue with either a single local thread if message order is important, or a pool of threads if you want to handle the incoming messages as quickly as possible.
Java has a built in set of core classes for building queues and starting threads to consume messages from those queues. Look at the classes in the java.util.concurrent package.
I am using a 3rd party blocking API. I am going to be using this API as follows:
while(true){
blockingAPI();
sendResultSomewhere();
}
blockingAPI() polls a server for a specific property until it gets a response.
In order to make things asynchronous to some extent I could spawn this API call within a separate thread. and have a callback implemented in Java to handle the response. I was wondering if I can use the netty framework in this scenario, and how I could do this? The examples I have seen involve a server that listens and communicates with a client, and I am not sure how my use case fits in.
If netty cannot be used, would my best bet be spawning a new thread and implementing a callback in Java?
Not sure what you really try to do:
Spawn internally a new thread: you could use LocalChannel with Netty to have intra-JVM process communication and therefore having something like you want, without any network consideration (only within the JVM). The blockingAPI will be computed within ServerLocalChannel side, while the result will be written once the client get back a response through the same LocalChannel.
Spawn but with a request from outside (network), then Netty could of course be used too there. Maybe still keeping a LocalChannel logic to separate network to compute.
Note that I could recommand to use asynchronous operation using LocalChannel (executing the blocking task), such that the send somewhere else is done without blocking the Netty's Network IO thread.
Network Handler side:
localChannel = creationWithinNetworkHandler(networkChannelCtx);
localChannel.writeAndFlush(something);
while LocalChannel handler server side could be as:
void read0(ChannelHandlerContext ctx, someData) {
blockingAPI();
ctx.channel().writeAndFlush(answear).addFutureListener(Channels.CloseFuture);
}
and LocalChannel handler client side could be as:
void read0(ChannelHandlerContext ctx, answear) {
//Using ctx from Network channel side
networkCtx.writeAndFlush(answear);
}
I currently have code which dispatches a request using the Ask Pattern. The dispatched request will generate an Akka Actor which sends a HTTP request and then returns the response. I'm using Akka's circuit breaker API to manage issues with the upstream web services i call.
If the circuitbreaker is in an open state then all subsequent requests are failing fast which is the desired effect. However when the actor fails fast it just throws a CircuitBreakerOpenException, stops the actor however control does not return to the code which made the initial request until an AskTimeoutException is generated.
This is the code which dispatches the request
Timeout timeout = new Timeout(Duration.create(10, SECONDS));
Future<Object> future = Patterns.ask(myActor, argMessage, timeout);
Response res = (Response ) Await.result(future, timeout.duration());
This is the circuitbreaker
getSender().tell(breaker.callWithSyncCircuitBreaker(new Callable<Obj>()
{
#Override
public Obj call() throws Exception {
return fetch(message);
}
}), getSelf()
);
getContext().stop(getSelf());
When this block of code is executed and if the circuit is open it fails fast throwing an exception however i want to return control back to the code which handles the future without having to wait for a timeout.
Is this possible?
When an actor fails out and is restarted, if it was processing a message, no response will be automatically sent to that sender. If you want to send that sender a message on that particular failure then catch that exception explicitly and respond back to that sender with a failed result, making sure to capture the sender first before you go into any future callbacks to avoid closing over this mutable state. You could also try to do this in the preRestart, but that's not very safe as by that time the sender might have changed if you are using futures inside the actor.
I've to manage a client-server-application with > 1K connected clients using netty 3.5.1. Sometimes updates get lost which are written to database when we disconnect our clients through restarting our servers. When performing a restart/shutdown, we shutdown our Netty components like this:
shutdown server-channel
disconnect all clients (via ChannelGroupFuture)
call releaseExternalResources() on our ChannelPipeline
call releaseExternalResources() on our ExecutionHandler which is part of our ChannelPipeline (is it necessary to invoke it manually?)
However I wonder why ExecutorUtil.terminate (which is called by the ExecutionHandler) does a shutdownNow on the passed ExecutorService, because shutdownNow drains all existing tasks in the queue and returns them. The tasks won't be executed because ExecutorUtil.terminate is of type void. Wouldn't it be more appropriate to invoke shutdown on the ExecutorService and wait for the completion?
That's a good suggestion.. Would you mind open a issue for it on our issue tracker[1]
[1 ]https://github.com/netty/netty/issues