I am using RxJava2 Flowables by subscribing to a stream of events from a PublishSubject.It's being used in enterprise level application and we don't have the choice of dropping any events.
I am using version RxJava 2.2.8
I am using BackpressureStrategy.BUFFER as I don't want to lose any of my events.
Also, I buffer again for 50000 or 3 minutes whichever is earlier. This I do as I want to consolidate events and then process them.
But I get the following errors in a few minutes of my run
io.reactivex.exceptions.MissingBackpressureException: Could not emit buffer due to lack of requests
at io.reactivex.internal.subscribers.QueueDrainSubscriber.fastPathOrderedEmitMax(QueueDrainSubscriber.java:121)
at io.reactivex.internal.operators.flowable.FlowableBufferTimed$BufferExactBoundedSubscriber.run(FlowableBufferTimed.java:569)
at io.reactivex.Scheduler$Worker$PeriodicTask.run(Scheduler.java:479)
at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
I tried increasing the buffer size by setting up, but there is no change in the behavior.
System.setProperty("rx2.buffer-size", "524288");
Also If I buffer for a longer time instead of 3 minutes, I get the exception after much longer time probably because my downstream performs better when the events are consolidated more. However, I don't have that choice because these are live events and needs processing immediately(in 3-5 minutes).
I also tried thread.sleep() before invoking the "subscription.next" in case of error but still getting the same results.
keySubject.hide()
.toFlowable(BackpressureStrategy.BUFFER)
.parallel()
.runOn(Schedulers.computation())
.map(e -> e.getContents())
.flatMap(s -> Flowable.fromIterable(s))
.sequential()
.buffer(3,TimeUnit.MINUTES,50000)
.subscribe(new Subscriber<List<String>>() {
#Override
public void onSubscribe(Subscription var1) {
innerSubscription = var1;
innerSubscription.request(1L);
}
#Override
public void onNext(List<String> logs) {
Subscription.request(1L);
/// Do some logic here
}
I want to know How do I handle the backpressure to avoid this exception? Is this exception because of ".buffer" method
Is there a way for me to check the status of these buffers. Also why even if I increase the rx2.buffer-size, I still get the exception in the same amount of time. Ideally, the system should run longer with a higher buffer size if the exception is because if buffer getting full.
Any help on the reason for this message "Could not emit buffer due to lack of requests at " will be great.
The thing is, why do you use a subject that isn't backpressure-aware? Are you using that as a poor man's event bus? Also, assuming e.getContents() is a simple getter I believe you can replace this whole block
.toFlowable(BackpressureStrategy.BUFFER)
.parallel()
.runOn(Schedulers.computation())
.map(e -> e.getContents())
.flatMap(s -> Flowable.fromIterable(s))
.sequential()
.buffer(3,TimeUnit.MINUTES,50000)
.subscribe(new Subscriber<List<String>>() { ... });
with
.flatMapIterable(e -> e.getContents())
.buffer(3,TimeUnit.MINUTES,50000)
.rebatchRequests(1)
.observeOn(Schedulers.computation())
.doOnNext(s -> /* Do some logic here */)
.subscribe();
Related
I am trying to implement a delay in a Spring Integration Flow.
I have one flow that is starting a process on another server and then I am checking after a delay if that process is completed or not.
When completed the flow should move to the next phase.
This seems to work it also shows in logs (and, clearly, in the flow itself), a long list of repetitions in the runexampleScriptWaiting channel.
I tried removing that channel change but then the flow gets stuck in that phase forever, never moving to completion.
How can I implement this so that a single runexampleScriptWaiting is shown / executed (something similar to a non-blocking while loop, I guess)?
I considered keeping it as is and just update my monitoring application (a very small frontend that shows which channels are in the payload's history) in order to get rid of duplicated channel lines but I also wondered if there is a better / more robust way to do this.
Here's a simplified example:
#Bean
public IntegrationFlow exampleIntegrationFlow() {
return IntegrationFlows
.from(exampleConfig.runexampleScript.get())
.<ExamplePayload>handle((payload, messageHeaders) -> examplePayloadService
.changeExampleServiceRequestStatus(payload, ExampleServiceStatus.STARTED))
.<ExamplePayload>handle(
(payload, messageHeaders) -> exampleScriptService.runexample(payload))
.channel(exampleConfig.runexampleScriptWaiting.get())
.<ExamplePayload, Boolean>route(jobStatusService::areJobsFinished,
router -> router
.subFlowMapping(true, exampleSuccessSubflow())
.subFlowMapping(false, exampleWaitSubflow())
.defaultOutputToParentFlow()
)
.get();
}
#Bean
public IntegrationFlow exampleWaitSubflow() {
return IntegrationFlows
.from(exampleConfig.runexampleScriptWaiting.get())
.<ExamplePayload>handle(
(payload, messageHeaders) -> {
interruptIgnoringSleep(1000);
return payload;
})
.channel(exampleConfig.runexampleScriptWaiting.get()) // Commenting this gets the process stuck
.get();
}
It is not clear what is your exampleConfig.runexampleScriptWaiting.get(), but what you have so far in the config is not OK. You have two subscribers to the same channel:
.channel(exampleConfig.runexampleScriptWaiting.get()) and the next route()
.from(exampleConfig.runexampleScriptWaiting.get()) and the next handle()
This may cause unexpected behavior, e.g. round-robin messages distribution.
I would do filter() and delay() instead in addition to an ExecutorChannel since you are asking about non-blocking retry:
.channel(exampleConfig.runexampleScriptWaiting.get())
.filter(jobStatusService::areJobsFinished,
filter -> filter.discardFlow(
discardFlow -> discardFlow
.delay(1000)
.channel(exampleConfig.runexampleScriptWaiting.get())))
The exampleSuccessSubflow could go just after this filter() as part of this flow or via to(exampleSuccessSubflow()).
Pay attention to that discardFlow: we delay non-finished message a little bit and produce it back to that runexampleScriptWaiting channel for calling this filter again. If you make this channel as an ExecutorChannel (or QueueChannel), your wait functionality is going to be non-blocking. But at the same time your main flow is still going to be blocked for this request since you continue waiting for reply. Therefore it might not make too much sense to make this filtering logic as non-blocking and you can still use that Thread.sleep() instead of delay().
The router solution also may work, but you cannot use that runexampleScriptWaiting channel as an input of that sub-flow. Probably that's the reason behind that your problem with "process stuck".
I need to send some data after user registered. I want to do first attempt in main thread, but if there are any errors, I want to retry 5 times with 10 minutes interval.
#Override
public void sendRegisterInfo(MailData data) {
Mono.just(data)
.doOnNext(this::send)
.doOnError(ex -> logger.warn("Main queue {}", ex.getMessage()))
.doOnSuccess(d -> logger.info("Send mail to {}", d.getRecipient()))
.onErrorResume(ex -> retryQueue(data))
.subscribe();
}
private Mono<MailData> retryQueue(MailData data) {
return Mono.just(data)
.delayElement(Duration.of(10, ChronoUnit.MINUTES))
.doOnNext(this::send)
.doOnError(ex -> logger.warn("Retry queue {}", ex.getMessage()))
.doOnSuccess(d -> logger.info("Send mail to {}", d.getRecipient()))
.retry(5)
.subscribe();
}
It works.
But I've got some questions:
Did I correct to make operation in doOnNext function?
Is it correct to use delayElement to make a delay between executions?
Did the thread blocked when waiting for delay?
And what the best practice to make a retries on error and make a delay between it?
doOnXXX for logging is fine. But for the actual element processing, you must prefer using flatMap rather than doOnNext (assuming your processing is asynchronous / can be converted to returning a Flux/Mono).
This is correct. Another way is to turn the code around and start from a Flux.interval, but here delayElement is better IMO.
The delay runs on a separate thread/scheduler (by default, Schedulers.parallel()), so not blocking the main thread.
There's actually a Retry builder dedicated to that kind of use case in the reactor-extra addon: https://github.com/reactor/reactor-addons/blob/master/reactor-extra/src/main/java/reactor/retry/Retry.java
I have a PublishSubject which receive emits from UI:
myPublishSubject
.map {
...
}
.doOnNext {
// using emitted item
}
.timeout (...) // wait for the gap!
.doOnNext {
// running a function after a specific gap between two item
}
.subscribe()
I want to wait a specific amount of time after last emit (not onComplete, cause it continue emitting later) and run a function. It can be interpreted as a gap between item emotion.
I am looking for something like timeout but this method issue is it kills the Observable with error.
You have to be a bit creative with publish and switchMap for example:
PublishSubject<Integer> ps = PublishSubject.create();
ps.publish(o ->
o.mergeWith(
o.switchMap(e ->
Observable.just(1).delay(200, TimeUnit.MILLISECONDS)
.ignoreElements()
.doOnCompleted(() -> System.out.println("Timeout action: " + e))
)
)
).subscribe(System.out::println);
ps.onNext(1);
ps.onNext(2);
Thread.sleep(100);
ps.onNext(3);
Thread.sleep(250);
ps.onNext(4);
Thread.sleep(250);
It works by sharing a source and routing into two ways, one is directly emitting while the other feeds a switchMap that when receives a new item, starts a delayed Observable and reacts to its completion (ignoring the original trigger element to avoid duplicate events due to mergeWith). When there is a new signal during the grace period, switchMap will cancel the previous delay and start with the newer delay.
I'm pretty new to RxJava and have some questions on patterns etc.
I'm creating an observable using the code below:
public Observable<Volume> getVolumeObservable(Epic epic) {
return Observable.create(event -> {
try {
listeners.add(streamingAPI.subscribeForChartCandles(epic.getName(), MINUTE, new HandyTableListenerAdapter() {
#Override
public void onUpdate(int i, String s, UpdateInfo updateInfo) {
if (updateInfo.getNewValue(CONS_END).equals(ONE)) {
event.onNext(new Volume(Integer.parseInt(updateInfo.getNewValue(LAST_TRADED_VOLUME))));
}
}
}));
} catch (Exception e) {
LOG.error("Error from volume observable", e);
}
});
}
Everything is working as expected, but I have some questions on error handling.
If I understand correctly, this is to be viewed as a "hot observble", i.e. events will happen regardless of there being a subscription or not (onUpdate is a callback used by a remote server which I have no control over).
I've chosen not to call onError here since I don't want the observable to stop emitting events in case of a single exception. Is there a better pattern to be used? .retry() comes to mind, but I'm not sure that it makes sense for a hot observable?
Also, how is the observable represented when the subscriptions is created, but before the first onNext is called? Is it just an Observable.empty()
1) Your observable is not hot. The distinguishing factor is whether multiple subscribers share the same subscription. Observable.create() invokes subscribe function for every subscriber, i.e. it is cold.
It is easy to make it hot though. Just add share() operator. It will subscribe with first subscriber and unsubscribe with last one. Do not forget to implement unsubscribe functionality with something like this:
event.setCancellable(() -> listeners.remove(...));
2) Errors could be recoverable and not recoverable.
In case you consider an error to be self-recoverable (no action required from your side) you should not call onError as this will kill your observable (no further events would be emitted). You can possibly notify your subscribers by emitting special Volume message with error details attached.
In case an error is fatal, e.g. you have failed to add listener, so there could be no further messages, you should not silently ignore this. Emit onError as your observable is not functional anyway.
In case an error requires actions from you, typically retry, or retry with timeout, you can add one of retryXxx() operators. Do this after create() but before share().
3) Observable is an object with subscribe() method. How exactly it is represented depends on the method you created it with. See source code of create() for example.
I'm trying to use RX Java to consume some data coming from a source that keeps sending objects.
I'm wondering how to implement a retry policy for cases in which my own code throws an exception. For example a network exception should trigger a retry with exponential backoff policy.
Some code :
message.map(this::processMessage)
.subscribe((message)->{
//do something after mapping
});
processMessage(message) is the method which contains the risky code that might fail and its the part of code that I want to retry but I dont want to stop the observable from consuming data from the source.
Any thoughts on this?
message
.map(this::processMessage)
.retryWhen(errors -> errors.flatMap(error -> {
if (error instanceof IOException) {
return Observable.just(null);
}
// For anything else, don't retry
return Observable.error(error);
})
.subscribe(
System.out::println,
error -> System.out.println("Error!")
);
or catch the error
message.map(this::processMessage)
.onErrorReturn(error -> "Empty result")
.subscribe((message)->{})
or procses the error
message
.map(this::processMessage)
.doOnError(throwable -> Log.e(TAG, "Throwable " + throwable.getMessage()))
.subscribe(
System.out::println,
error -> System.out.println("Error!")
);
Untested, but retryWhen differs to repeatWhen that is not only called in onComplete.
http://blog.danlew.net/2016/01/25/rxjavas-repeatwhen-and-retrywhen-explained/
-> Each error is flatmapped so that we can either return onNext(null) (to trigger a resubscription) or onError(error) (to avoid resubscription).
Backoff Policy:
source.retryWhen(errors ->
errors
.zipWith(Observable.range(1, 3), (n, i) -> i)
.flatMap(retryCount -> Observable.timer((long) Math.pow(5, retryCount), TimeUnit.SECONDS))
);
flatMap + timer is preferable over delay in this case because it lets us modify the delay by the number of retries. The above retries three times and delays each retry by 5 ^ retryCount, giving you exponential backoff with just a handful of operators!
Take an example from articles:
https://medium.com/#v.danylo/server-polling-and-retrying-failed-operations-with-retrofit-and-rxjava-8bcc7e641a5a#.a6ll8d5bt
http://kevinmarlow.me/better-networking-with-rxjava-and-retrofit-on-android/
They helped me oneday.
Recently I developed the library that exactly suits your needs.
RetrofitRxErrorHandler
If you combine Exponential strategy with backupObservable you will get the expected result.