I'm using a non Blocking (Async) sending message to Kafka using this :
ListenableFuture<SendResult<Integer, String>> future = template.send(record);
future.addCallback(new ListenableFutureCallback<SendResult<Integer, String>>() {
#Override
public void onSuccess(SendResult<Integer, String> result) {
handleSuccess(data);
}
#Override
public void onFailure(Throwable ex) {
handleFailure(data, record, ex);
}
});
This work perfectly when the send action does its work.
But when there is a connection problem (server down for example), the result become non asynchronous and the method remains blocked until the end of the duration of max.block.ms.
This is natural in Async KAfka producer. You have two options
Either reduce the max.block.ms but don't reduce it too much.
You can wait for acks
You can also create a callback function for onCompletion()
Related
I am working on a spring boot app where I am receiving a callback from a 3rd party api and that 3rd party also expects a response.
There are multiple db operations that I need to perform and to avoid data inconsistency during load I am using a SQS queue to push data and do all the operation.
The Thing is I also want to send the response back to the 3rd party immediately and the DB operations will be running in the background.
How is this possible with the SQS queue and CompletableFuture?
Here's the snippet:
// this method is a SQS Listener.
public PaymentCallback acceptPaymentForPendingOrders(String callbackRequest) {
PaymentCallbackRequest paymentCallbackRequest = decryptCallBackResponse(callbackRequest);
// here's all the DB operations I am doing async.
CompletableFuture<Void> completableFuture = CompletableFuture.runAsync(new Runnable() {
#Override
#Transactional
public void run() {
try {
orderSettlingRequest.setPaymentId(savedPayment.getId());
ResponseEntity response = orderServiceClient.settleBtoBOrders(orderSettlingRequest);
} catch (Exception e) {
//credit the full amount to wallet of the user
log.error("Error occurred while settling b2b orders using credit",e);
}
}
});
//sending response immediately not waiting for asnyc operations.
return paymentCallbackRepo.save(paymentCallback);
}
Can I do this when using queue? Is this feasible?
I am performing an asynchronous operation which returns a future object in a loop (say 10 messages) .
As I understand callback method is automatically triggered and executed when the Future completes its task.
Suppose my 7th future is in pending stage.How can i complete this future exceptionally?
And what is the best way to handle this kind of scenario.
List<ListenableFuture<SendResult<String, String>>> cf = new ArrayList<ListenableFuture<SendResult<String, String>>>();
future = kafkaTemplate.send(topicName, message);
cf.add(future);
i++;
future.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
#Override
public void onSuccess(SendResult<String, String> result) {
syso("sent success");
}
#Override
public void onFailure(Throwable ex) {
System.out.println(" sending failed");
}
});
Why would you want to do that?
If there is an exception, the kafka-clients will call the template's callback with the exception and the template will complete the future exceptionally.
If you really need to do it for some reason (but I'd like to understand why) you can cast it to a SettableListenerFuture.
i'm trying to implement pub sub pattern using grpc but i'm confusing a bit about how to do it properly.
my proto: rpc call (google.protobuf.Empty) returns (stream Data);
client:
asynStub.call(Empty.getDefaultInstance(), new StreamObserver<Data>() {
#Override
public void onNext(Data value) {
// process a data
#Override
public void onError(Throwable t) {
}
#Override
public void onCompleted() {
}
});
} catch (StatusRuntimeException e) {
LOG.warn("RPC failed: {}", e.getStatus());
}
Thread.currentThread().join();
server service:
public class Sender extends DataServiceGrpc.DataServiceImplBase implements Runnable {
private final BlockingQueue<Data> queue;
private final static HashSet<StreamObserver<Data>> observers = new LinkedHashSet<>();
public Sender(BlockingQueue<Data> queue) {
this.queue = queue;
}
#Override
public void data(Empty request, StreamObserver<Data> responseObserver) {
observers.add(responseObserver);
}
#Override
public void run() {
while (!Thread.currentThread().isInterrupted()) {
try {
// waiting for first element
Data data = queue.take();
// send head element
observers.forEach(o -> o.onNext(data));
} catch (InterruptedException e) {
LOG.error("error: ", e);
Thread.currentThread().interrupt();
}
}
}
}
How to remove clients from global observers properly? How to received some sort of a signal when connection drops?
How to manage client-server reconnections? How to force client reconnect when connection drops?
Thanks in advance!
In the implementation of your service:
#Override
public void data(Empty request, StreamObserver<Data> responseObserver) {
observers.add(responseObserver);
}
You need to get the Context of the current request, and listen for cancellation. For single-request, multi-response calls (a.k.a. Server streaming) the gRPC generated code is simplified to pass in the the request directly. This means that you con't have direct access to the underlying ServerCall.Listener, which is how you would normally listen for clients disconnecting and cancelling.
Instead, every gRPC call has a Context associated with it, which carries the cancellation and other request-scoped signals. For your case, you just need to listen for cancellation by adding your own listener, which then safely removes the response observer from your linked hash set.
As for reconnects: gRPC clients will automatically reconnect if the connection is broken, but usually will not retry the RPC unless it is safe to do so. In the case of server streaming RPCs, it is usually not safe to do, so you'll need to retry the RPC on your client directly.
I need a functionality that would allow to push asynchronously messages to my PublishSubject and to process them at a certain pace (actually one by one) via a ConnectableObservable. Unfortunately it seems that the call to onNext of the PublishSubject is not released until the underlying Subscriber processes the message.
It takes good few seconds to process each message and in debug mode I see that it executes before invocation of the method that pushes the message to PublishSubject is removed from stack - "After push..." always appear in console after internal logs inside the Subscriber...
So I have this RestEndpoint:
#PUT
#Path("{id}")
#TokenAuthenticated
public Response postResource(#PathParam(value="id") final String extId) {
executorService.execute(new Runnable() {
#Override
public void run() {
try {
Message metadata = processor.apply(extId);
log.info("Before push...");
dataImporter.pushData(metadata);
log.info("After push...");
} catch (Exception e) {
e.printStackTrace();
}
}
});
return Response.ok("Request received successfully").build();
}
Here's the constructor of the DataImporter:
public DataImporter(final String configFile) {
dataToImportSubject = PublishSubject.create();
dataToImportObservable = dataToImportSubject.publish();
dataToImportObservable.connect();
dataToImportObservable
.onBackpressureBuffer(1, new Action0() {
#Override
public void call() {
logger.debug("Buffer full...");
}
})
.subscribeOn(Schedulers.io())
.subscribe(new Subscriber<Message>() {
#Override
public void onCompleted() {
// TODO Auto-generated method stub
}
#Override
public void onError(Throwable e) {
logger.error("Error importing "+e.getMessage());
}
#Override
public void onNext(Message value) {
request(1);
importResult(configFile, value);
}
#Override
public void onStart() {
request(1);
}
});
}
Then pushData of DataImporter is just pushing to PublishSubject's onNext method..:
public void pushData(Message metadata) {
dataToImportSubject.onNext(metadata);
}
And here're the declaration of PublishSubject and ConnectableObservable:
public class DataImporter implements ImporterProxy{
private final PublishSubject<Message> dataToImportSubject;
private final ConnectableObservable<Message> dataToImportObservable;
PublishSubjects emit to their consumers on the thread of the original onXXX call:
JavaDocs
Scheduler:
PublishSubject does not operate by default on a particular Scheduler and the Observers get notified on the thread the respective onXXX methods were invoked.
You have to move the processing to some other thread with observeOn because the observeOn can move the onXXX calls to another thread.
subscribeOn does not have any practical effect on Subjects in general because it only affects the subscription thread, and won't modulate the subsequent onXXX calls to those subjects.
RxJava, by default, is synchronous. You need to introduce operators into your observer chain to perform actions on other threads. When you read the documentation on each operator in Observable, you will see statements like "... does not operator on a particular scheduler" -- this indicates that data flows through that operator synchronously.
To get an observer chain to perform actions on other threads, you can use an operator like subscribeOn() with a scheduler to have operations performed on that scheduler. In your example, you likely will want to use Schedulers.io() to provide a background thread.
I'm planning to use Netty to design a TCP Server. When the client connects, I have to immediately start pumping
XML data to the client continuously...for hours/days. Its that simple.
So, I override "channelConnected" method and send data from that method, right?...thats great.
I will be using the following ChannelFactory
ChannelFactory factory =
new NioServerSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool());
NioServerSocketChannelFactory documentation says
A worker thread performs non-blocking read and write for one or more Channels in a non-blocking mode.
Good.
According to effective Java Item 51: Don't depend on the thread scheduler, I want the worker thread to do a "unit of work" and then finish/return.
So in my case, though I have to send data continuously, I want to send some chunk (lets say 1 MB) and then be done (unit of work completed), so that worker thread can return. Then I'll send another 1 MB.
Below example is from the official guide of Netty HERE.
I guess the question is then, in this scenario, if i had to unconditionally keep sending time to the client, how would I do it, considering
each send as a unit of work.
One way of doing it would be to just put a while loop and do a Thread.Sleep. Any other way?
package org.jboss.netty.example.time;
public class TimeServerHandler extends SimpleChannelHandler {
#Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e) {
Channel ch = e.getChannel();
ChannelBuffer time = ChannelBuffers.buffer(4);
time.writeInt(System.currentTimeMillis() / 1000);
ChannelFuture f = ch.write(time);
f.addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) {
Channel ch = future.getChannel();
ch.close();
}
});
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) {
e.getCause().printStackTrace();
e.getChannel().close();
}
}
Doing a while/sleep would work, but would not be in the Netty super-scalable style. It would be thread-per-connection programming.
Instead, schedule a periodic job on an Executor that writes a message to the channel.