I need a functionality that would allow to push asynchronously messages to my PublishSubject and to process them at a certain pace (actually one by one) via a ConnectableObservable. Unfortunately it seems that the call to onNext of the PublishSubject is not released until the underlying Subscriber processes the message.
It takes good few seconds to process each message and in debug mode I see that it executes before invocation of the method that pushes the message to PublishSubject is removed from stack - "After push..." always appear in console after internal logs inside the Subscriber...
So I have this RestEndpoint:
#PUT
#Path("{id}")
#TokenAuthenticated
public Response postResource(#PathParam(value="id") final String extId) {
executorService.execute(new Runnable() {
#Override
public void run() {
try {
Message metadata = processor.apply(extId);
log.info("Before push...");
dataImporter.pushData(metadata);
log.info("After push...");
} catch (Exception e) {
e.printStackTrace();
}
}
});
return Response.ok("Request received successfully").build();
}
Here's the constructor of the DataImporter:
public DataImporter(final String configFile) {
dataToImportSubject = PublishSubject.create();
dataToImportObservable = dataToImportSubject.publish();
dataToImportObservable.connect();
dataToImportObservable
.onBackpressureBuffer(1, new Action0() {
#Override
public void call() {
logger.debug("Buffer full...");
}
})
.subscribeOn(Schedulers.io())
.subscribe(new Subscriber<Message>() {
#Override
public void onCompleted() {
// TODO Auto-generated method stub
}
#Override
public void onError(Throwable e) {
logger.error("Error importing "+e.getMessage());
}
#Override
public void onNext(Message value) {
request(1);
importResult(configFile, value);
}
#Override
public void onStart() {
request(1);
}
});
}
Then pushData of DataImporter is just pushing to PublishSubject's onNext method..:
public void pushData(Message metadata) {
dataToImportSubject.onNext(metadata);
}
And here're the declaration of PublishSubject and ConnectableObservable:
public class DataImporter implements ImporterProxy{
private final PublishSubject<Message> dataToImportSubject;
private final ConnectableObservable<Message> dataToImportObservable;
PublishSubjects emit to their consumers on the thread of the original onXXX call:
JavaDocs
Scheduler:
PublishSubject does not operate by default on a particular Scheduler and the Observers get notified on the thread the respective onXXX methods were invoked.
You have to move the processing to some other thread with observeOn because the observeOn can move the onXXX calls to another thread.
subscribeOn does not have any practical effect on Subjects in general because it only affects the subscription thread, and won't modulate the subsequent onXXX calls to those subjects.
RxJava, by default, is synchronous. You need to introduce operators into your observer chain to perform actions on other threads. When you read the documentation on each operator in Observable, you will see statements like "... does not operator on a particular scheduler" -- this indicates that data flows through that operator synchronously.
To get an observer chain to perform actions on other threads, you can use an operator like subscribeOn() with a scheduler to have operations performed on that scheduler. In your example, you likely will want to use Schedulers.io() to provide a background thread.
Related
I have a service in spring which needs to fetch data using ten different methods.
I would like to have these methods execute parallelly to do some DB operations and return to the parent thread. But the parent thread should wait until all the responses come and then return a response.
In my current approach, I am using reactive mono to execute all methods asynchronously but the main thread is not waiting for the subscriber methods to finish.
Below are my two methods which I have subscribed
private Mono<BaseResponse> getProfileDetails(long profileId){
return new Mono<BaseResponse>() {
#Override
public void subscribe(Subscriber<? super BaseResponse> s) {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
// DB Operation
System.out.println("Inside getProfileDetails");
s.onNext(new BaseResponse());
}
};
}
private Mono<Address> getAddressDetails(long profileId){
return new Mono<Address>() {
#Override
public void subscribe(Subscriber<? super Address> s) {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
// DB Operation
System.out.println("Inside getAddressDetails");
s.onNext(new Address());
}
};
}
And below is my main method
public BaseResponse getDetails(long profileId){
ExecutorService executors = Executors.newFixedThreadPool(2);
Mono<BaseResponse> profileDetail = this.getProfileDetails(profileId).subscribeOn(Schedulers.fromExecutor(executors));
Mono<BaseResponse> addressDetail = this.getAddressDetails(profileId).subscribeOn(Schedulers.fromExecutor(executors));
List<BaseResponse> list = new ArrayList<>();
profileDetail.mergeWith(addressDetail)
.subscribe(consumer -> {
list.add(consumer);
});
System.out.println("list: "+new Gson().toJson(list));
executors.shutdown();
return response;
}
Below is my output:
list: []
Inside getProfileDetails
Inside getAddressDetails
My output shows that the main thread is not waiting for the subscriber to finish its task,
so how can I handle this situation?
I'm assuming your getProfileDetails() and getAddressDetails() methods are just placeholders, as they don't make much sense as written.
That being said, if this is your entire application here, and you genuinely just want to block before completing, you may as well just change your current subscribe() call for a doOnNext(), then just blockLast():
profileDetail.mergeWith(addressDetail)
.doOnNext(consumer -> {
list.add(consumer);
})
.blockLast();
Blocking on reactive threads is usually ill-advised in reactive applications for good reason, but in this case you literally just want to block before exiting outright - so I can't see much downside here.
I am writing some tests for some rest api, i have a dispatcher that diapctch my rest request to a WebClient from vertx, in some cases i want to wait for the response from the rest api to return before i can continue further with my assertions, the code that dispatch the requests is wrapped inside other classes, so i am not directly making those requests from the tests, i have 2 implementations from my request dispatcher,one for production and one for tests, the tests dispatcher looks like this :
public class TestRequestDispatcher extends AbstractRequestDispatcher {
#Override
protected void dispatchRequest(ServerRequest request, ServerRequestEventFactory requestEventFactory) {
request.getSender()
.send(request,
new ServerRequestCallBack() {
#Override
public <T> void onSuccess(T response) {
requestEventFactory.makeSuccess(request, response).fire();
}
#Override
public void onFailure(FailedResponseBean failedResponse) {
requestEventFactory.makeFailed(request, failedResponse).fire();
}
});
}
}
this should then call some code that builds a WebClient and call its send method to send the request to the server.
And in order to wait for the response i decided to use the CountDownLatch and modified my code to the following
public class TestRequestDispatcher extends AbstractRequestDispatcher {
#Override
protected void dispatchRequest(ServerRequest request, ServerRequestEventFactory requestEventFactory) {
CountDownLatch requestWait = new CountDownLatch(1);
request.getSender()
.send(request,
new ServerRequestCallBack() {
#Override
public <T> void onSuccess(T response) {
requestWait.countDown();
requestEventFactory.makeSuccess(request, response).fire();
}
#Override
public void onFailure(FailedResponseBean failedResponse) {
requestWait.countDown();
requestEventFactory.makeFailed(request, failedResponse).fire();
}
});
try {
requestWait.await(20, TimeUnit.SECONDS);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
I am using a large timeout here to make sure that the response should return before the timeout is up, so what happens is that i can breakpoint and see the WebCLient.send method being called, and then it pauses at requestWait.wait(...) but the callbacks are never invoked until the CountDownLatch time out is up. while i was expecting the WebClient to send the request and when ever the response is returned it will invoke the callbacks which in return will count down and break the wait before the timwout is up.
Testing with a normal thread things seems to work, i created some runnable class with some sleep period ..less that the CountDownTime latch. like the the following
public class SenderWorker implements Runnable {
private CountDownLatch countDownLatch;
public SenderWorker(CountDownLatch countDownLatch) {
this.countDownLatch = countDownLatch;
}
#Override
public void run() {
try {
Thread.sleep(5000L);
countDownLatch.countDown();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
then in the dispatcher :
public class TestRequestDispatcher extends AbstractRequestDispatcher {
#Override
protected void dispatchRequest(ServerRequest request, ServerRequestEventFactory requestEventFactory) {
CountDownLatch requestWait = new CountDownLatch(1);
new Thread(new SenderWorker(requestWait))
.start();
try {
requestWait.await(20, TimeUnit.SECONDS);
System.out.println("i am here");
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
And this works.. it calls the run method..the sleeps, then call the requestWait.wait(..) and after 5 seconds it breaks out of the wait.
I tried to execute the code that calls the WebClient inside an executeBlocking and also tried the runOnContext and even tried to run it inside a thread just like how i did with the SenderWorker but still the same result .. the WebClient is blocked until the timeout is up.
Any idea what i am doing wrong here and how i can make this works.?!
You may want to consider vertx-unit or vertx-junit5 for testing asynchronous code with Vert.x.
Other than that asynchronous operations should be composed rather than spinning threads and waiting for count-down latches. Vert.x offers several options to do that:
callbacks chaining
future composition
RxJava (1 and 2)
Kotlin coroutines
Quasar.
I don't understand when AsyncListener#onError method is called.
Javadoc doesn't help:
Notifies this AsyncListener that an asynchronous operation has failed to complete.
How could it fail? How can I reproduce this error?
UPDATE:
// in HttpServlet doGet method
AsyncContext asyncContext = req.startAsync();
asyncContext.addListener(new AsyncListener() {
// some code
#Override
public void onError(AsyncEvent event) {
// when is it called?
}
});
executeInSomeOtherThread(asyncContext);
What do I need to do in other thread to fail this async operation?
onError will be called if there was an Exception thrown while carrying out the asynchronous operation.
They are typically Throwables that extend java.io.IOException caused by I/O failures because of an unreliable connection or protocol level exceptions due to a logical error because of a mismatch between the client and the server.
You can get the Throwable when onError is invoked by calling:
event.getThrowable();
EDIT to address mjaggard's follow-on questions
Forgetting about AsyncContext for a second, consider the following class:
public class MyRunnableClass implements Runnable {
private Listener mListener;
interface Listener {
void onError(Throwable error);
}
public void setListener(Listener listener) {
mListener = listener;
}
#Override
public void run() {
// Some important code to be executed goes here
// Pretend an exception was caught in a try/catch/finally
// block that was doing an I/O operation
Throwable someError = new IOException();
if (mListener != null) {
mListener.onError(someError);
}
}
}
Is it more clear now how the listener's onError method will be invoked because an exception was raised when MyRunnableClass's run method was invoked?
MyRunnableClass mrc = new MyRunnableClass();
mrc.setListener(new Listener() {
#Override
public void onError(Throwable error) {
}
});
Executors.newSingleThreadScheduledExecutor().schedule(mrc, 1000L, TimeUnit.MILLISECONDS);
This is no different from how an AsyncContext holds onto a listener and notifies it if it encounters an exception that it wishes to report to the listener. How the run method gets invoked is really secondary to the fact that the owner of the code being executed is also the one that holds a reference to the listener.
I need advice on the following:
I have a #Scheduled service method which has a fixedDelay of a couple of seconds in which it does scanning of a work queue and processing of apropriate work if it finds any. In the same service I have a method which puts work in the work queue and I would like this method to imediately trigger scanning of the queue after it's done (since I'm sure that there will now be some work to do for the scanner) in order to avoid the delay befor the scheduled kicks in (since this can be seconds, and time is somewhat critical).
An "trigger now" feature of the Task Execution and Scheaduling subsystem would be ideal, one that would also reset the fixedDelay after execution was initiated maually (since I dont want my manual execution to collide with the scheduled one). Note: work in the queue can come from external source, thus the requirement to do periodic scanning.
Any advice is welcome
Edit:
The queue is stored in a document-based db so local queue-based solutions are not appropriate.
A solution I am not quite happy with (don't really like the usage of raw threads) would go something like this:
#Service
public class MyProcessingService implements ProcessingService {
Thread worker;
#PostCreate
public void init() {
worker = new Thread() {
boolean ready = false;
private boolean sleep() {
synchronized(this) {
if (ready) {
ready = false;
} else {
try {
wait(2000);
} catch(InterruptedException) {
return false;
}
}
}
return true;
}
public void tickle() {
synchronized(this) {
ready = true;
notify();
}
}
public void run() {
while(!interrupted()) {
if(!sleep()) continue;
scan();
}
}
}
worker.start();
}
#PreDestroy
public void uninit() {
worker.interrup();
}
public void addWork(Work work) {
db.store(work);
worker.tickle();
}
public void scan() {
List<Work> work = db.getMyWork();
for (Work w : work) {
process();
}
}
public void process(Work work) {
// work processing here
}
}
Since the #Scheduled method wouldn't have any work to do if there are no items in the work-queue, that is, if no one put any work in the queue between the execution cycles. On the same note, if some work-item was inserted into the work-queue (by an external source probably) immediately after the scheduled-execution was complete, the work won't be attended to until the next execution.
In this scenario, what you need is a consumer-producer queue. A queue in which one or more producers put in work-items and a consumer takes items off the queue and processes them. What you want here is a BlockingQueue. They can be used for solving the consumer-producer problem in a thread-safe manner.
You can have one Runnable that performs the tasks performed by your current #Scheduled method.
public class SomeClass {
private final BlockingQueue<Work> workQueue = new LinkedBlockingQueue<Work>();
public BlockingQueue<Work> getWorkQueue() {
return workQueue;
}
private final class WorkExecutor implements Runnable {
#Override
public void run() {
while (true) {
try {
// The call to take() retrieves and removes the head of this
// queue,
// waiting if necessary until an element becomes available.
Work work = workQueue.take();
// do processing
} catch (InterruptedException e) {
continue;
}
}
}
}
// The work-producer may be anything, even a #Scheduled method
#Scheduled
public void createWork() {
Work work = new Work();
workQueue.offer(work);
}
}
And some other Runnable or another class might put in items as following:
public class WorkCreator {
#Autowired
private SomeClass workerClass;
#Override
public void run() {
// produce work
Work work = new Work();
workerClass.getWorkQueue().offer(work);
}
}
I guess that's the right way to solve the problem you have at hand. There are several variations/configurations that you can have, just look at the java.util.concurrent package.
Update after question edited
Even if the external source is a db, it is still a producer-consumer problem. You can probably call the scan() method whenever you store data in the db, and the scan() method can put the data retrieved from the db into the BlockingQueue.
To address the actual thing about resetting the fixedDelay
That is not actually possible, wither with Java, or with Spring, unless you handle the scheduling part yourself. There is no trigger-now functionality as well. If you have access to the Runnable that's doing the task, you can probably call the run() method yourself. But that would be the same as calling the processing method yourself from anywhere and you don't really need the Runnable.
Another possible workaround
private Lock queueLock = new ReentrantLock();
#Scheduled
public void findNewWorkAndProcess() {
if(!queueLock.tryLock()) {
return;
}
try {
doWork();
} finally {
queueLock.unlock();
}
}
void doWork() {
List<Work> work = getWorkFromDb();
// process work
}
// To be called when new data is inserted into the db.
public void newDataInserted() {
queueLock.lock();
try {
doWork();
} finally {
queueLock.unlock();
}
}
the newDataInserted() is called when you insert any new data. If the scheduled execution is in progress, it will wait until it is finished and then do the work. The call to lock() here is blocking since we know that there is some work in the database and the scheduled-call might have been called before the work was inserted. The call to acquire lock in findNewWorkAndProcess() in non-blocking as, if the lock has been acquired by the newDataInserted method, it would mean that the scheduled method shouldn't be executed.
Well, you can fine tune as you like.
My current code uses series of asynchronous processes that culminate in results. I need to wrap each of these in such a way that each is accessed by a synchronous method with the result as a return value. I want to use executor services to do this, so as to allow many of these to happen at the same time. I have the feeling that Future might be pertinent to my implementation, but I can't figure out a good way to make this happen.
What I have now:
public class DoAJob {
ResultObject result;
public void stepOne() {
// Passes self in for a callback
otherComponent.doStepOne(this);
}
// Called back by otherComponent once it has completed doStepOne
public void stepTwo(IntermediateData d) {
otherComponent.doStepTwo(this, d);
}
// Called back by otherComponent once it has completed doStepTwo
public void stepThree(ResultObject resultFromOtherComponent) {
result = resultFromOtherComponent;
//Done with process
}
}
This has worked pretty well internally, but now I need to map my process into a synchronous method with a return value like:
public ResultObject getResult(){
// ??? What goes here ???
}
Does anyone have a good idea about how to implement this elegantly?
If you want to turn an asynchronous operation (which executes a callback when finished), into a synchronous/blocking one, you can use a blocking queue. You can wrap this up in a Future object if you wish.
Define a blocking queue which can hold just one element:
BlockingQueue<Result> blockingQueue = new ArrayBlockingQueue<Result>(1);
Start your asynchronous process (will run in the background), and write the callback such that when it's done, it adds its result to the blocking queue.
In your foreground/application thread, have it take() from the queue, which blocks until an element becomes available:
Result result = blockingQueue.take();
I wrote something similar before (foreground thread needs to block for an asynchronous response from a remote machine) using something like a Future, you can find example code here.
I've done something similar with the Guava library; these links might point you in the right direction:
Is it possible to chain async calls using Guava?
https://code.google.com/p/guava-libraries/wiki/ListenableFutureExplained
If you like to get your hands dirty, you can do this
ResultObject result;
public void stepOne()
otherComponent.doStepOne(this);
synchronized(this)
while(result==null) this.wait();
return result;
public void stepThree(ResultObject resultFromOtherComponent)
result = resultFromOtherComponent;
synchronized(this)
this.notify();
Or you can use higher level concurrency tools, like BlockingQueue, Semaphore, CountdownLatch, Phaser, etc etc.
Note that DoAJob is not thread safe - trouble ensured if two threads call stepOne at the same time.
I recommend using invokeAll(..). It will submit a set of tasks to the executor, and block until the last one completes (successfully/with exception). It then returns a list of completed Future objects, so you can loop on them and merge the results into a single ResultObject.
In you wish to run only a single task in a synchronous manner, you can use the following:
executor.invokeAll(Collections.singleton(task));
--edit--
Now I think I understand better your needs. I assume that you need a way to submit independent sequences of tasks. Please take a look at the code I posted in this answer.
Bumerang is my async only http request library which is constructed for Android http requests using Java -> https://github.com/hanilozmen/Bumerang . I needed to make synchronous calls without touching my library. Here is my complete code. npgall's answer inspired me, thanks! Similar approach would be applied to all kinds of async libraries.
public class TestActivity extends Activity {
MyAPI api = (MyAPI) Bumerang.get().initAPI(MyAPI.class);
BlockingQueue<Object> blockingQueue = new ArrayBlockingQueue<Object>(1);
static int indexForTesting;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_test);
Thread t = new Thread(new Runnable() {
#Override
public void run() {
for(int i = 0; i < 10; i++) {
getItems();
try {
Object response = blockingQueue.take(); // waits for the response
Log.i("TAG", "index " + indexForTesting + " finished. Response " + response.toString());
} catch (Exception e) {
e.printStackTrace();
}
}
}
});
t.start();
}
void getItems() {
Log.i("TAG", "index " + ++indexForTesting + " started");
api.getItems(new ResponseListener<Response<List<ResponseModel>>>() {
#Override
public void onSuccess(Response<List<ResponseModel>> response) {
List<ResponseModel> respModel = response.getResponse();
try {
blockingQueue.put(response);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
#Override
public void onError(Response<List<ResponseModel>> response) {
Log.i("onError", response.toString());
try {
blockingQueue.put(response);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
}
}