We Keep Coding

SmallRye Mutiny unable to process events asynchronously using subscription

I am developing an application that returns Multi<String>, I would like to make some modifications to it, so I have added some methods, but for some reason it does not enter the next method at all.
My other methods are working absolutely fine. Because I am able to collect it and add it to a List, but I want to do some execution asynchronously, so using this approach.
private final ManagedExecutor managedExecutor;
public void writeTo(StreamingInfo streamingInfo) {
streamingInfo
.getEvents()
.runSubscriptionOn(managedExecutor)
.subscribe()
.withSubscriber(
new Subscriber < String > () {
#Override
public void onSubscribe(Subscription s) {
System.out.println("OnSubscription Method");
System.out.println("ON SUBS END");
}
#Override
public void onNext(String event) {
System.out.println("On Next Method");
}
#Override
public void onError(Throwable t) {
System.out.println("OnError Method");
}
#Override
public void onComplete() {
System.out.println("On Complete Method");
}
});
}
I get the following output:
OnSubscription Method
ON SUBS END
Which means that your subscription is not working for some reason. If I do not add subscription and directly collect to List then everything works as expected. Can anyone suggest what am I doing wrong here?

This is because the underlying Reactive Streams specification that SmallRye Mutiny implements has a built-in backpressure mechanism. The client (in your case your subscriber) needs to request the next item manually from the producer (events) otherwise, no item is sent down the reactive pipeline.
You need to save the Subscription object you receive in the onSubscribe method and call its request(long) method when you can process next item(s):
.withSubscriber(
new Subscriber<String>() {
private Subscription subscription;
#Override
public void onSubscribe(Subscription s) {
System.out.println("OnSubscription Method");
System.out.println("ON SUBS END");
subscription = s;
subscription.request(1);
}
#Override
public void onNext(String event) {
System.out.println("On Next Method");
subscription.request(1);
}
#Override
public void onError(Throwable t) {
System.out.println("OnError Method");
}
#Override
public void onComplete() {
System.out.println("On Complete Method");
}
});
In SmallRye there is also an easier way to do this:
events
.onSubscription()
.invoke(() -> {
System.out.println("OnSubscription Method");
System.out.println("ON SUBS END");
})
.onItem()
.invoke(event -> System.out.println("On Next Method"))
.onFailure()
.invoke(t -> System.out.println("OnError Method"))
.onCompletion()
.invoke(() -> System.out.println("On Complete Method"))
.subscribe()
.with(value -> {});

Reactivestreams Subscriber not works with Spring Reactor Mono. Why?

I have a reactor.core.publisher.Mono variable and want to subscribe to org.reactivestreams.Subscriber, though it seems not works. I cannot get Why onNext method never called? I see onSubscribe method called fine. I could be mistaken, but as Mono implements Publisher, subscriber should work. Right?
#Test
public void subscriberTest() {
Mono<String> m = Mono.just("Hello!");
Subscriber<String> s = new Subscriber<String>() {
#Override
public void onSubscribe(Subscription s) {
System.out.println("Subscription "+s);
}
#Override
public void onNext(String t) {
System.out.println("onNext "+t);
}
#Override
public void onError(Throwable t) {
System.out.println("Throwable "+t);
}
#Override
public void onComplete() {
System.out.println("onComplete");
}
};
m.subscribe(s);
Mono<String> m1 = Mono.just("Bye!");
m1.subscribe(System.out::println);
}
Though the variable m1 subscription with method reference works fine. Here console output:
Subscription reactor.core.publisher.StrictSubscriber#4b168fa9
Bye!
Here I expect to see Hello! phrase too.

https://www.reactive-streams.org/reactive-streams-1.0.0-javadoc/org/reactivestreams/Subscriber.html#onSubscribe-org.reactivestreams.Subscription- Here it is stated that No data will start flowing until Subscription.request(long) is invoked.

Creating a Flowable that emits items at a limited rate to avoid the need to buffer events

I've got a data access object that passes each item in a data source to a consumer:
public interface Dao<T> {
void forEachItem(Consumer<T> item);
}
This always produces items in a single threaded way - I can't currently change this interface.
I wanted to create a Flowable from this interface:
private static Flowable<String> flowable(final Dao dao) {
return Flowable.create(emitter -> {
dao.forEachItem(item ->
emitter.onNext(item));
emitter.onComplete();
}, ERROR);
}
If I use this Flowable in a situation where the processing takes longer than the rate at which items are emitted then I understandably get a missing back pressure exception as I am using ERROR mode:
Dao<String> exampleDao =
itemConsumer ->
IntStream.range(0, 1_000).forEach(i ->
itemConsumer.accept(String.valueOf(i)));
flowable(exampleDao)
.map(v -> {
Thread.sleep(100);
return "id:" + v;
})
.blockingSubscribe(System.out::println);
I don't wish to buffer items - seems like this could lead to exhausting memory on very large data sets - if the operation is significantly slower than the producer.
I was hoping there would be a backpressure mode that would allow the emitter to block when passed next/completion events when it detects back pressure but that does not seem to be the case?
In my case as I know that the dao produces items in a single threaded way I thought I would be able to do something like:
dao.forEachItem(item -> {
while (emitter.requested() == 0) {
waitABit();
}
emitter.onNext(item)
});
but this seems to hang forever.
How wrong is my approach? :-) Is there a way of producing items in a way that respects downstream back pressure given my (relatively restrictive) set of circumstances?
I know I could do this with a separate process writing to a queue and then write a Flowable based on consuming from that queue- would that be the preferred approach instead?

Check the part of the Flowable, especially the part with Supscription.request(long). I hope that gets you on the right way.
The TestProducerfrom this example produces Integerobjects in a given range and pushes them to its Subscriber. It extends the Flowable<Integer> class. For a new subscriber, it creates a Subscription object whose request(long) method is used to create and publish the Integer values.
It is important for the Subscription that is passed to the subscriber that the request() method which calls onNext()on the subscriber can be recursively called from within this onNext() call. To prevent a stack overflow, the shown implementation uses the outStandingRequests counter and the isProducing flag.
class TestProducer extends Flowable<Integer> {
static final Logger logger = LoggerFactory.getLogger(TestProducer.class);
final int from, to;
public TestProducer(int from, int to) {
this.from = from;
this.to = to;
}
#Override
protected void subscribeActual(Subscriber<? super Integer> subscriber) {
subscriber.onSubscribe(new Subscription() {
/** the next value. */
public int next = from;
/** cancellation flag. */
private volatile boolean cancelled = false;
private volatile boolean isProducing = false;
private AtomicLong outStandingRequests = new AtomicLong(0);
#Override
public void request(long n) {
if (!cancelled) {
outStandingRequests.addAndGet(n);
// check if already fulfilling request to prevent call between request() an subscriber .onNext()
if (isProducing) {
return;
}
// start producing
isProducing = true;
while (outStandingRequests.get() > 0) {
if (next > to) {
logger.info("producer finished");
subscriber.onComplete();
break;
}
subscriber.onNext(next++);
outStandingRequests.decrementAndGet();
}
isProducing = false;
}
}
#Override
public void cancel() {
cancelled = true;
}
});
}
}
The Consumer in this example extends DefaultSubscriber<Integer> and on start and after consuming an Integer requests the next one. On consuming the Integer values, there is a little delay, so the backpressure will be built up for the producer.
class TestConsumer extends DefaultSubscriber<Integer> {
private static final Logger logger = LoggerFactory.getLogger(TestConsumer.class);
#Override
protected void onStart() {
request(1);
}
#Override
public void onNext(Integer i) {
logger.info("consuming {}", i);
if (0 == (i % 5)) {
try {
Thread.sleep(500);
} catch (InterruptedException ignored) {
// can be ignored, just used for pausing
}
}
request(1);
}
#Override
public void onError(Throwable throwable) {
logger.error("error received", throwable);
}
#Override
public void onComplete() {
logger.info("consumer finished");
}
}
in the following main method of a test class the producer and consumer are created and wired up:
public static void main(String[] args) {
try {
final TestProducer testProducer = new TestProducer(1, 1_000);
final TestConsumer testConsumer = new TestConsumer();
testProducer
.subscribeOn(Schedulers.computation())
.observeOn(Schedulers.single())
.blockingSubscribe(testConsumer);
} catch (Throwable t) {
t.printStackTrace();
}
}
When running the example, the logfile shows that the consumer runs continuously, while the producer only gets active when the internal Flowable buffer of rxjava2 needs to be refilled.

Observable is not asynchronous

I am learning RxJava and am testing a scenario where I read data from a DB and then post it to a Queue. I just made a sample mock of the whole process but I don't seem to find the Observable working as I wanted it to ie. asynchronously.
This is my code:
package rxJava;
import java.util.ArrayList;
import java.util.List;
import rx.Observable;
import rx.Observer;
import rx.functions.Action1;
public class TestClass {
public static void main(String[] args) {
TestClass test = new TestClass();
System.out.println("---START---");
test.getFromDB().subscribe(new Observer<String>() {
#Override
public void onCompleted() {
System.out.println("Publish complete.");
}
#Override
public void onError(Throwable t) {
System.out.println(t.getMessage());
}
#Override
public void onNext(String s) {
test.publishToQueue(s).subscribe(new Observer<Boolean>() {
#Override
public void onNext(Boolean b) {
if (b) {
System.out.println("Successfully published.");
}
}
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable arg0) {
}
});
};
});
System.out.println("---END---");
}
public Observable<String> getFromDB() {
List<String> list = new ArrayList<String>();
for (int i = 0; i < 30; i++) {
list.add(Integer.toString(i));
}
return Observable.from(list).doOnNext(new Action1<String>() {
#Override
public void call(String temp) {
if (temp.contains("2")) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
});
}
public Observable<Boolean> publishToQueue(String s) {
return Observable.defer(() -> {
try {
if (s.contains("7")) {
Thread.sleep(700);
}
System.out.println("Published:: " + s);
} catch (InterruptedException e) {
e.printStackTrace();
}
return Observable.just(true);
});
}
}
Suppose I get a list from the DB asynchronously and want to post it to the queue,. I have used an Observable returned from getFromDB and have subscribed to it which mimics the data I get from DB. Every time I get the data from DB , I want to push it to a queue using publishToQueue which also returns an Observable. I wanted to make the queue call also asynchronous. Now on positive acknowledgement from the queue such as the Boolean which I am returning (Observable<Boolean>), I want to print something.
So basically I just want both the processes to be asynchronous. For every data from DB, I push it to the Queue asynchronously.
I have added Thread.sleep() in both the methods, db call and queue so as to mimic a delay and to test the asynchronous operations. I think this is what causing the problem. But I also tried Obseravable.delay() but that doesn't even produce any output.
Please help me understand how this works and how I can make it work as I want it to.

You have to specified subscribeOn value.
Observable.just("one", "two", "three", "four", "five")
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(/* an Observer */);
http://reactivex.io/documentation/operators/subscribeon.html

By default, RxJava is synchronous. It means that everything will be perform in the same thread (and the current thread), by default. You can perform tasks in another thread thanks to observeOn / subscribeOn methods, or using some operators that perform tasks in another job (because it use another scheduler, like delay, interval, ...)
In your example, you have to explitly set in which scheduler the subscription will pe performed. (here, in which thread Observable.from will emit your list)
test.getFromDb()
.subscribeOn(Schedulers.io())
.subscribe();
Then you can use the flatMap operator and calling your publishToQueue method. This method will be executed in the previous scheduler, but you can force it to use another scheduler, thanks to observeOn method. Everything after the observeOn method will be executed in another thread.
test.fromDb()
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.computation())
.flatMap(l -> test.publishToqueue(l))
.subscribe();

RxJava Backpressure (Fast producer slow consumer)

i have execution method which does some time consuming network calls on io thread
example
/**
* network call
* #param value
* #return
*/
private Observable<Integer> execute(final int value) {
return Observable.create(new Observable.OnSubscribe<Integer>() {
#Override
public void call(Subscriber<? super Integer> subscriber) {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("done " + value);
subscriber.onNext(value);
subscriber.onCompleted();
}
}).subscribeOn(Schedulers.io());
}
then i have list of "commands" which must be executed in order. (one after another)
example (Observable.range(x,y) represents list of commands)
public List<Integer> testObservableBackpressure(){
return Observable.range(0,5).flatMap(new Func1<Integer, Observable<Integer>>() {
#Override
public Observable<Integer> call(Integer integer) {
System.out.println("started " + integer);
return exeute(integer);
}
}).toList().toBlocking().single();
}
in this way out put is
started 0
started 1
started 2
started 3
started 4
done 0
done 1
done 2
done 4
done 3
Produses faster than is consuming
I want results like that
started 0
done 0
started 1
done 1
started 2
done 2
...
but..
public List<Integer> testObservableBackpressure(){
return Observable.create(new Observable.OnSubscribe<Integer>() {
#Override
public void call(final Subscriber<? super Integer> subscriber) {
Observable.range(0,5).subscribe(new Subscriber<Integer>() {
#Override
public void onStart() {
request(1);
}
#Override
public void onCompleted() {
subscriber.onCompleted();
}
#Override
public void onError(Throwable e) {
subscriber.onError(e);
}
#Override
public void onNext(Integer integer) {
System.out.println("started " + integer);
execute(integer).subscribe(new Action1<Integer>() {
#Override
public void call(Integer integer) {
subscriber.onNext(integer);
request(1);
}
});
}
});
}
}).toList().toBlocking().single();
}
this way results are as expected
started 0
done 0
started 1
done 1
started 2
done 2
started 3
done 3
started 4
My question would be if there is another more elegant way to handle this problem?

I'm not sure you need any particular backpressure strategy here. Just use concatMap.
If you use concatMap instead of flatMap, each new input value will only subscribe when the last Observable emitted from the concatMap completes. Under the hood, concatMap uses a SerialSubscription for this. That should give you the ordering you want.

The output that I get when I run your code is:
started 0
started 1
started 2
started 3
started 4
done 1
done 3
done 4
done 2
done 0
Notice that the "done" messages are out of order. This is due to the fact that your code basically parallelises the execution of each call to execute. For each item emitted by the Observable.range you flatMap an Observable that runs on its own IOScheduler. Thus every item is handled in parallel on separate threads which makes it impossible for the items to stay in order and correctly interleaved. One option to achieve the desired behaviour is to make sure that all items run on the same IOScheduler (instead of each item on its own):
import rx.Observable;
import rx.Subscriber;
import rx.functions.Func1;
import rx.schedulers.Schedulers;
import java.util.List;
public class Test {
private Observable<Integer> execute(final int value) {
return Observable.create(new Observable.OnSubscribe<Integer>() {
#Override
public void call(Subscriber<? super Integer> subscriber) {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("done " + value);
subscriber.onNext(value);
subscriber.onCompleted();
}
});
}
public List<Integer> testObservableBackpressure(){
return Observable.range(0, 5).flatMap(new Func1<Integer, Observable<Integer>>() {
#Override
public Observable<Integer> call(Integer integer) {
System.out.println("started " + integer);
return execute(integer);
}
}).subscribeOn(Schedulers.io()).toList().toBlocking().single();
}
public static void main(String[] args) {
new Test().testObservableBackpressure();
}
}
Notice that the only difference is where you invoke the subscribeOn operator. This code results in the following output:
started 0
done 0
started 1
done 1
started 2
done 2
started 3
done 3
started 4
done 4