We Keep Coding

Mono.when firing before publishers complete

Sample code:
Flux<Integer> fluxSrc = Flux.<Integer> create(e -> {
e.next(1);
try {
Thread.sleep(500);
} catch (InterruptedException e1) {
throw new RuntimeException(e1);
}
e.complete();
})
.publishOn(Schedulers.single())
.publish().autoConnect(2);
Flux<Integer> fluxA = fluxSrc
.publishOn(Schedulers.single())
.map(j -> 10 + j);
fluxA.subscribe(System.out::println);
Mono<Integer> monoB = fluxSrc
.publishOn(Schedulers.single())
.reduce(20, (j, k) -> {
try {
Thread.sleep(1000);
} catch (InterruptedException e1) {
throw new RuntimeException(e1);
}
return j + k;
});
monoB.subscribe(System.out::println);
Mono.when(fluxA, monoB)
.block();
System.out.println("After");
This produces the following output:
11
After
21
Why does it not wait for both publishers (fluxA and monoB) to complete? How should I structure the code so I make sure all publishers complete before After is reached?

By using .publish(), fluxSrc is turned into hot flux. Consider:
Hot publishers, on the other hand, do not depend on any number of
subscribers. They might start publishing data right away and would
continue doing so whenever a new Subscriber comes in (in which case
said subscriber would only see new elements emitted after it
subscribed). For hot publishers, something does indeed happen before
you subscribe.
(https://projectreactor.io/docs/core/release/reference/#reactor.hotCold)
One way to fix it is to get rid of publish and operate on cold stream. Another one is to change .autoConnect(2); to .autoConnect(3); - that's because you want to start processing data when 3rd subscription - Mono.when(fluxA, monoB).block(); is reached (previous ones are fluxA.subscribe and monoB.subscribe).
Edit:
When did wait for sources to finish, but it got onComplete signal from previous subsription.
What probably happened is:
flux A was subsribed by fluxA.subscribe(System.out::println);, emitted 11 and printed it.
flux B was subsribed by monoB.subscribe(System.out::println); and started reduction.
Mono.when was subsribed (which triggered "multicasting" - fluxes were subsribed second time).
Reduction started, it's result will be 21.
Another reduction started and was immediately finished with result 20 (reducing empty stream - only item from fluxSrc was already consumed by another reduction).
flux A sent onComplete to both subsribers.
flux B sent onComplete with result of reduction = 20. It was passed to subscription made by Mono.when, that's why it wasn't printed.
Both fluxes sent onComplete since Mono.when subsription, so that After was printed.
Around that time first reduction was completed with value 21, which was passed to monoB.subscribe(System.out::println);

How to create blocking backpressure with rxjava Flowables?

I have a Flowable that we are returning in a function that will continually read from a database and add it to a Flowable.
public void scan() {
Flowable<String> flow = Flowable.create((FlowableOnSubscribe<String>) emitter -> {
Result result = new Result();
while (!result.hasData()) {
result = request.query(skip, limit);
partialResult.getResult()
.getFeatures().forEach(feature -> emmitter.emit(feature));
}
}, BackpressureStrategy.BUFFER)
.subscribeOn(Schedulers.io());
return flow;
}
Then I have another object that can call this method.
myObj.scan()
.parallel()
.runOn(Schedulers.computation())
.map(feature -> {
//Heavy Computation
})
.sequential()
.blockingSubscribe(msg -> {
logger.debug("Successfully processed " + msg);
}, (e) -> {
logger.error("Failed to process features because of error with scan", e);
});
My heavy computation section could potentially take a very long time. So long in fact that there is a good chance that the database requests will load the whole database into memory before the consumer finishes the first couple entries.
I have read up on backpressure with rxjava but the only 4 options essentially make me drop data or replace it with the last.
Is there a way to make it so that when I call emmitter.emit(feature) the call blocks until there is more room in the Flowable?
I.E I want to treat the Flowable as a blocking queue where push will sleep if the queue is past the capacity.

Converting an Observable to a Flowable with backpressure in RxJava2

I am observing the lines produced by a NetworkResource, wrapping it in an Observable.create. Here is the code, missing try/catch and cancellation for simplicity:
fun linesOf(resource: NetworkResource): Observable<String> =
Observable.create { emitter ->
while (!emitter.isDisposed) {
val line = resource.readLine()
Log.i(TAG, "Emitting: $line")
emitter.onNext(line)
}
}
The problem is that later I want to turn it into a Flowable using observable.toFlowable(LATEST) to add backpressure in case my consumer can't keep up, but depending on how I do it, the consumer stops receiving items after item 128.
A) this way everything works:
val resource = ...
linesOf(resource)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.toFlowable(BackpressureStrategy.LATEST)
.subscribe { Log.i(TAG, "Consuming: $it") }
B) here the consumer gets stuck after 128 items (but the emitting continues):
val resource = ...
linesOf(resource)
.toFlowable(BackpressureStrategy.LATEST)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe { Log.i(TAG, "Consuming: $it") } // <-- stops after 128
In option A) everything works without any issues, and I can see the Emitting: ... log side by side with the Consuming: ... log.
In option B) I can see the Emitting: ... log message happily emitting new lines, but I stop seeing the Consuming: ... log message after item 128, even though the emitting continues.
Question: Can someone help me understand why this happens?

First of all, you are using the wrong type and wrong operator. Using Flowable removes the need for conversion. Using Flowable.generate gets you backpressure:
Flowable.generate(emitter -> {
String line = resource.readLine();
if (line == null) {
emitter.onComplete();
} else {
emitter.onNext(line);
}
});
Second, the reason your version hangs is due to a same pool deadlock caused by subscribeOn. Requests from downstream are scheduled behind your eager emission loop and can not take effect, stopping the emission at the default 128 elements. Use Flowable.subscribeOn(scheduler, false) to avoid this case.

Proper termination of a stuck Couchbase Observable

I'm trying to delete a batch of couchbase documents in rapid fashion according to some constraint (or update the document if the constraint isn't satisfied). Each deletion is dubbed a "parcel" according to my terminology.
When executing, I run into a very strange behavior - the thread in charge of this task starts working as expected for a few iterations (at best). After this "grace period", couchbase gets "stuck" and the Observable doesn't call any of its Subscriber's methods (onNext, onComplete, onError) within the defined period of 30 seconds.
When the latch timeout occurs (see implementation below), the method returns but the Observable keeps executing (I noticed that when it kept printing debug messages when stopped with a breakpoint outside the scope of this method).
I suspect couchbase is stuck because after a few seconds, many Observables are left in some kind of a "ghost" state - alive and reporting to their Subscriber, which in turn have nothing to do because the method in which they were created has already finished, eventually leading to java.lang.OutOfMemoryError: GC overhead limit exceeded.
I don't know if what I claim here makes sense, but I can't think of another reason for this behavior.
How should I properly terminate an Observable upon timeout? Should I? Any other way around?
public List<InfoParcel> upsertParcels(final Collection<InfoParcel> parcels) {
final CountDownLatch latch = new CountDownLatch(parcels.size());
final List<JsonDocument> docRetList = new LinkedList<JsonDocument>();
Observable<JsonDocument> obs = Observable
.from(parcels)
.flatMap(parcel ->
Observable.defer(() ->
{
return bucket.async().get(parcel.key).firstOrDefault(null);
})
.map(doc -> {
// In-memory manipulation of the document
return updateDocs(doc, parcel);
})
.flatMap(doc -> {
boolean shouldDelete = ... // Decide by inner logic
if (shouldDelete) {
if (doc.cas() == 0) {
return Observable.just(doc);
}
return bucket.async().remove(doc);
}
return (doc.cas() == 0 ? bucket.async().insert(doc) : bucket.async().replace(doc));
})
);
obs.subscribe(new Subscriber<JsonDocument>() {
#Override
public void onNext(JsonDocument doc) {
docRetList.add(doc);
latch.countDown();
}
#Override
public void onCompleted() {
// Due to a bug in RxJava, onError() / retryWhen() does not intercept exceptions thrown from within the map/flatMap methods.
// Therefore, we need to recalculate the "conflicted" parcels and send them for update again.
while(latch.getCount() > 0) {
latch.countDown();
}
}
#Override
public void onError(Throwable e) {
// Same reason as above
while (latch.getCount() > 0) {
latch.countDown();
}
}
};
);
latch.await(30, TimeUnit.SECONDS);
// Recalculating remaining failed parcels and returning them for another cycle of this method (there's a loop outside)
}

I think this is indeed due to the fact that using a countdown latch doesn't signal the source that the flow of data processing should stop.
You could use more of rxjava, by using toList().timeout(30, TimeUnit.SECONDS).toBlocking().single() instead of collecting in an (un synchronized and thus unsafe) external list and of using the countdownLatch.
This will block until a List of your documents is returned.

When you create your couchbase env in code, set computationPoolSize to something large. When the Couchbase clients runs out of threads using async it just stops working, and wont ever call the callback.

RxJava. Initial onNext when subscription starts?

I'm trying to implement a class that emits changes using an Observable.
When a subscription is done to this observable I want to send an starting/initialization event. Then I want to send the usual events.
For example. Lets say I have two different subscribers A and B. A and B starts subscribing at different times. If MyClass.getChanges() emits event no. 1,2,3,4 and 5.
If A starts it subscription between event 1,2 then it should receive the following events:
InitialEvent, 2, 3, 4, 5.
If B starts it subscription between event 4 and 5, then B should receive the following events:
InitialEvent, 5.
How to do this using RxJava?
Thanks!
Edit 1
I think I need to explain that the "InitialEvent" is different each time it's emitted. It's calculated by MyClass each time a new subscriber starts to subscribe from getChanged().
My scenario is that MyClass contains a list. The "initialEvent" contains the list at the moment when the subscription is done. Then each change to this list is emitted from getChanges().

Sorry to post this 2 years later, but I had the same need and found this question unanswered.
What I did is the following:
public Observable<Event> observe() {
return Observable.defer(() ->
subject.startWith(createInitialEvent())
);
}
The idea is the following:
defer() executes the passed-in lambda expression when an observer subscribes to the Observable returned by the method observe(). So basically, it executes subject.startWith(...), which returns an Observable that is the actual source of event for the subscriber.
subject.startWith(...) emits an initial event (specified by startWith(...)) followed by those emitted by the subject.
So, if I come back to the original post:
if an observer starts it subscription between event 1,2 then it should receive the following events: InitialEvent, 2, 3, 4, 5.

What you're looking for is PublishSubject. Subjects are hot Observables, in that they do not wait for Observers to subscribe to them before beginning to emit their items. Here's a bit of info on Subjects.
Here's a quick demo of your use-case
PublishSubject<String> subject = PublishSubject.create();
Observable<String> InitEvent = Observable.just("init");
Observable<String> A = subject.asObservable();
Observable<String> B = subject.asObservable();
subject.onNext("1");
A.startWith(InitEvent)
.subscribe(s -> System.out.println("A: " + s));
subject.onNext("2");
subject.onNext("3");
subject.onNext("4");
B.startWith(InitEvent)
.subscribe(s -> System.out.println("B: " + s));
subject.onNext("5");

Possibly not really elegant way how about just using a flag? It looks like you just want to replace the first emitted event.
e.g. for one subscription the following logic:
boolean firstTimeA = true;
myCustomObservable.subscribe(s -> {
System.out.println(firstTimeA ? "initEvent" : s.toString());
if(firstTimeA) firstTimeA = false;
});
And since you want to have a second subscription just create a firstTimeB and update it your B subscription.

If I understand what you are asking something like this should work for you
int last = 0;
Observable obs;
List<Integer> list = new ArrayList<>();
public SimpleListObservable() {
obs = Observable.create(new Observable.OnSubscribe<Integer>() {
#Override
public void call(Subscriber<? super Integer> subscriber) {
while(last < 30) {
last++;
list.add(last);
subscriber.onNext(last);
}
subscriber.onCompleted();
}
});
}
public Observable<Integer> process() {
return Observable.from(list).concatWith(obs);
}
As the source observable collects values they are added to the List (you can transform the items as you see fit, filter them out, etc) and then when ObserverB subscribes it will get a replay of the items already collected in the List before continuing with the source observable output.
This simple test should demonstrate the outcome
public void testSequenceNext() {
final SimpleListObservable obs = new SimpleListObservable();
final Observer<Integer> ob2 = Mockito.mock(Observer.class);
obs.process().subscribe(new Observer<Integer>() {
#Override
public void onCompleted() {
ob1Complete = true;
}
#Override
public void onError(Throwable e) {
e.printStackTrace();
}
#Override
public void onNext(Integer integer) {
System.out.println("ob1: " + integer);
if (integer == 20) {
obs.process().subscribe(ob2);
}
}
});
ArgumentCaptor<Integer> captor = ArgumentCaptor.forClass(Integer.class);
Mockito.verify(ob2, Mockito.times(30)).onNext(captor.capture());
for (Integer value : captor.getAllValues()) {
System.out.println(value);
}
}

What do you think of this, I've made part of my API of course as I'm on a phone :
public class StreamOfSomething {
new StreamOfSomething() {
// source of events like
events = Observable.range(0, 1_000_000_000)
.doOnNext(set::add) // some operation there
.map(Event::change)
.publish()
.refCount();
}
public Observable<Event> observeChanges() {
return events.startWith(
Observable.just(Event.snapshot(set))); // start stream with generated event
}
}
And the client can do something like :
Observable.timer(2, 4, TimeUnit.SECONDS)
.limit(2)
.flatMap(t -> theSourceToWatch.observeChanges().limit(10))
.subscribe(System.out::println);
Note however if you are in a multithreaded environment you may have to synchronize when you are subscribing to block any modification, otherwise the list may change before it get's emitted. Or rework this class completely around observables, I don't know yet how to achieve this though.