i have execution method which does some time consuming network calls on io thread
example
/**
* network call
* #param value
* #return
*/
private Observable<Integer> execute(final int value) {
return Observable.create(new Observable.OnSubscribe<Integer>() {
#Override
public void call(Subscriber<? super Integer> subscriber) {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("done " + value);
subscriber.onNext(value);
subscriber.onCompleted();
}
}).subscribeOn(Schedulers.io());
}
then i have list of "commands" which must be executed in order. (one after another)
example (Observable.range(x,y) represents list of commands)
public List<Integer> testObservableBackpressure(){
return Observable.range(0,5).flatMap(new Func1<Integer, Observable<Integer>>() {
#Override
public Observable<Integer> call(Integer integer) {
System.out.println("started " + integer);
return exeute(integer);
}
}).toList().toBlocking().single();
}
in this way out put is
started 0
started 1
started 2
started 3
started 4
done 0
done 1
done 2
done 4
done 3
Produses faster than is consuming
I want results like that
started 0
done 0
started 1
done 1
started 2
done 2
...
but..
public List<Integer> testObservableBackpressure(){
return Observable.create(new Observable.OnSubscribe<Integer>() {
#Override
public void call(final Subscriber<? super Integer> subscriber) {
Observable.range(0,5).subscribe(new Subscriber<Integer>() {
#Override
public void onStart() {
request(1);
}
#Override
public void onCompleted() {
subscriber.onCompleted();
}
#Override
public void onError(Throwable e) {
subscriber.onError(e);
}
#Override
public void onNext(Integer integer) {
System.out.println("started " + integer);
execute(integer).subscribe(new Action1<Integer>() {
#Override
public void call(Integer integer) {
subscriber.onNext(integer);
request(1);
}
});
}
});
}
}).toList().toBlocking().single();
}
this way results are as expected
started 0
done 0
started 1
done 1
started 2
done 2
started 3
done 3
started 4
My question would be if there is another more elegant way to handle this problem?
I'm not sure you need any particular backpressure strategy here. Just use concatMap.
If you use concatMap instead of flatMap, each new input value will only subscribe when the last Observable emitted from the concatMap completes. Under the hood, concatMap uses a SerialSubscription for this. That should give you the ordering you want.
The output that I get when I run your code is:
started 0
started 1
started 2
started 3
started 4
done 1
done 3
done 4
done 2
done 0
Notice that the "done" messages are out of order. This is due to the fact that your code basically parallelises the execution of each call to execute. For each item emitted by the Observable.range you flatMap an Observable that runs on its own IOScheduler. Thus every item is handled in parallel on separate threads which makes it impossible for the items to stay in order and correctly interleaved. One option to achieve the desired behaviour is to make sure that all items run on the same IOScheduler (instead of each item on its own):
import rx.Observable;
import rx.Subscriber;
import rx.functions.Func1;
import rx.schedulers.Schedulers;
import java.util.List;
public class Test {
private Observable<Integer> execute(final int value) {
return Observable.create(new Observable.OnSubscribe<Integer>() {
#Override
public void call(Subscriber<? super Integer> subscriber) {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("done " + value);
subscriber.onNext(value);
subscriber.onCompleted();
}
});
}
public List<Integer> testObservableBackpressure(){
return Observable.range(0, 5).flatMap(new Func1<Integer, Observable<Integer>>() {
#Override
public Observable<Integer> call(Integer integer) {
System.out.println("started " + integer);
return execute(integer);
}
}).subscribeOn(Schedulers.io()).toList().toBlocking().single();
}
public static void main(String[] args) {
new Test().testObservableBackpressure();
}
}
Notice that the only difference is where you invoke the subscribeOn operator. This code results in the following output:
started 0
done 0
started 1
done 1
started 2
done 2
started 3
done 3
started 4
done 4
Related
In a simple demo program I compared the output of Mono.just().map().subscribe() and Mono.just().flatMap().subscribe(), to my surprise map() is executed after onSubscribe() but flatMap() is before.
The program:
public static void main(#Nonnull String[] args) {
// 1. Case with map()
Mono.just(1).map(number -> {
System.out.printf("[%s] mapping:%d%n", threadName(), number);
return number * 10;
}).subscribe(buildSubscriber());
// 2. Case with flatMap()
System.out.println();
Mono.just(1).flatMap(number -> {
System.out.printf("[%s] flat-mapping:%d%n", threadName(), number);
return Mono.just(number * 10);
}).subscribe(buildSubscriber());
}
#SuppressWarnings("ReactiveStreamsSubscriberImplementation")
#Nonnull
private static Subscriber<Integer> buildSubscriber() {
return new Subscriber<>() {
private Subscription subscription;
#Override
public void onSubscribe(Subscription subscription) {
System.out.printf("[%s] onSubscribe%n", threadName());
this.subscription = subscription;
subscription.request(1);
}
#Override
public void onNext(Integer item) {
System.out.printf("[%s] onNext:%s%n", threadName(), item);
subscription.request(1);
}
#Override
public void onError(Throwable throwable) {
System.out.printf("[%s] onError:%s%n", threadName(), throwable);
}
#Override
public void onComplete() {
System.out.printf("[%s] onComplete%n", threadName());
}
};
}
#Nonnull
private static String threadName() {
return Thread.currentThread().getName();
}
The output:
[main] onSubscribe
[main] mapping:1
[main] onNext:10
[main] onComplete
[main] flat-mapping:1
[main] onSubscribe
[main] onNext:10
[main] onComplete
In the output of the first case onSubscribe is before mapping:1. It's conceivable that the subscriber made a request for one event and then the publisher produced one, and then this very event went through the transformation and other operations if any.
In the output of the second case flat-mapping:1 comes first, which means onSubscribe() is not yet called but the event is emitted already. The documentation says that map() is implemented by applying a synchronous function, and flatMap() transforms asynchronously. What we witness here is that everything happens in the main thread.
Things seem in contradiction, neither flatMap() is performed asynchronously, nor the publisher emits on subscriber's demand, so what's going on under the hood in reactor？
I've got a data access object that passes each item in a data source to a consumer:
public interface Dao<T> {
void forEachItem(Consumer<T> item);
}
This always produces items in a single threaded way - I can't currently change this interface.
I wanted to create a Flowable from this interface:
private static Flowable<String> flowable(final Dao dao) {
return Flowable.create(emitter -> {
dao.forEachItem(item ->
emitter.onNext(item));
emitter.onComplete();
}, ERROR);
}
If I use this Flowable in a situation where the processing takes longer than the rate at which items are emitted then I understandably get a missing back pressure exception as I am using ERROR mode:
Dao<String> exampleDao =
itemConsumer ->
IntStream.range(0, 1_000).forEach(i ->
itemConsumer.accept(String.valueOf(i)));
flowable(exampleDao)
.map(v -> {
Thread.sleep(100);
return "id:" + v;
})
.blockingSubscribe(System.out::println);
I don't wish to buffer items - seems like this could lead to exhausting memory on very large data sets - if the operation is significantly slower than the producer.
I was hoping there would be a backpressure mode that would allow the emitter to block when passed next/completion events when it detects back pressure but that does not seem to be the case?
In my case as I know that the dao produces items in a single threaded way I thought I would be able to do something like:
dao.forEachItem(item -> {
while (emitter.requested() == 0) {
waitABit();
}
emitter.onNext(item)
});
but this seems to hang forever.
How wrong is my approach? :-) Is there a way of producing items in a way that respects downstream back pressure given my (relatively restrictive) set of circumstances?
I know I could do this with a separate process writing to a queue and then write a Flowable based on consuming from that queue- would that be the preferred approach instead?
Check the part of the Flowable, especially the part with Supscription.request(long). I hope that gets you on the right way.
The TestProducerfrom this example produces Integerobjects in a given range and pushes them to its Subscriber. It extends the Flowable<Integer> class. For a new subscriber, it creates a Subscription object whose request(long) method is used to create and publish the Integer values.
It is important for the Subscription that is passed to the subscriber that the request() method which calls onNext()on the subscriber can be recursively called from within this onNext() call. To prevent a stack overflow, the shown implementation uses the outStandingRequests counter and the isProducing flag.
class TestProducer extends Flowable<Integer> {
static final Logger logger = LoggerFactory.getLogger(TestProducer.class);
final int from, to;
public TestProducer(int from, int to) {
this.from = from;
this.to = to;
}
#Override
protected void subscribeActual(Subscriber<? super Integer> subscriber) {
subscriber.onSubscribe(new Subscription() {
/** the next value. */
public int next = from;
/** cancellation flag. */
private volatile boolean cancelled = false;
private volatile boolean isProducing = false;
private AtomicLong outStandingRequests = new AtomicLong(0);
#Override
public void request(long n) {
if (!cancelled) {
outStandingRequests.addAndGet(n);
// check if already fulfilling request to prevent call between request() an subscriber .onNext()
if (isProducing) {
return;
}
// start producing
isProducing = true;
while (outStandingRequests.get() > 0) {
if (next > to) {
logger.info("producer finished");
subscriber.onComplete();
break;
}
subscriber.onNext(next++);
outStandingRequests.decrementAndGet();
}
isProducing = false;
}
}
#Override
public void cancel() {
cancelled = true;
}
});
}
}
The Consumer in this example extends DefaultSubscriber<Integer> and on start and after consuming an Integer requests the next one. On consuming the Integer values, there is a little delay, so the backpressure will be built up for the producer.
class TestConsumer extends DefaultSubscriber<Integer> {
private static final Logger logger = LoggerFactory.getLogger(TestConsumer.class);
#Override
protected void onStart() {
request(1);
}
#Override
public void onNext(Integer i) {
logger.info("consuming {}", i);
if (0 == (i % 5)) {
try {
Thread.sleep(500);
} catch (InterruptedException ignored) {
// can be ignored, just used for pausing
}
}
request(1);
}
#Override
public void onError(Throwable throwable) {
logger.error("error received", throwable);
}
#Override
public void onComplete() {
logger.info("consumer finished");
}
}
in the following main method of a test class the producer and consumer are created and wired up:
public static void main(String[] args) {
try {
final TestProducer testProducer = new TestProducer(1, 1_000);
final TestConsumer testConsumer = new TestConsumer();
testProducer
.subscribeOn(Schedulers.computation())
.observeOn(Schedulers.single())
.blockingSubscribe(testConsumer);
} catch (Throwable t) {
t.printStackTrace();
}
}
When running the example, the logfile shows that the consumer runs continuously, while the producer only gets active when the internal Flowable buffer of rxjava2 needs to be refilled.
I am learning RxJava and am testing a scenario where I read data from a DB and then post it to a Queue. I just made a sample mock of the whole process but I don't seem to find the Observable working as I wanted it to ie. asynchronously.
This is my code:
package rxJava;
import java.util.ArrayList;
import java.util.List;
import rx.Observable;
import rx.Observer;
import rx.functions.Action1;
public class TestClass {
public static void main(String[] args) {
TestClass test = new TestClass();
System.out.println("---START---");
test.getFromDB().subscribe(new Observer<String>() {
#Override
public void onCompleted() {
System.out.println("Publish complete.");
}
#Override
public void onError(Throwable t) {
System.out.println(t.getMessage());
}
#Override
public void onNext(String s) {
test.publishToQueue(s).subscribe(new Observer<Boolean>() {
#Override
public void onNext(Boolean b) {
if (b) {
System.out.println("Successfully published.");
}
}
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable arg0) {
}
});
};
});
System.out.println("---END---");
}
public Observable<String> getFromDB() {
List<String> list = new ArrayList<String>();
for (int i = 0; i < 30; i++) {
list.add(Integer.toString(i));
}
return Observable.from(list).doOnNext(new Action1<String>() {
#Override
public void call(String temp) {
if (temp.contains("2")) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
});
}
public Observable<Boolean> publishToQueue(String s) {
return Observable.defer(() -> {
try {
if (s.contains("7")) {
Thread.sleep(700);
}
System.out.println("Published:: " + s);
} catch (InterruptedException e) {
e.printStackTrace();
}
return Observable.just(true);
});
}
}
Suppose I get a list from the DB asynchronously and want to post it to the queue,. I have used an Observable returned from getFromDB and have subscribed to it which mimics the data I get from DB. Every time I get the data from DB , I want to push it to a queue using publishToQueue which also returns an Observable. I wanted to make the queue call also asynchronous. Now on positive acknowledgement from the queue such as the Boolean which I am returning (Observable<Boolean>), I want to print something.
So basically I just want both the processes to be asynchronous. For every data from DB, I push it to the Queue asynchronously.
I have added Thread.sleep() in both the methods, db call and queue so as to mimic a delay and to test the asynchronous operations. I think this is what causing the problem. But I also tried Obseravable.delay() but that doesn't even produce any output.
Please help me understand how this works and how I can make it work as I want it to.
You have to specified subscribeOn value.
Observable.just("one", "two", "three", "four", "five")
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(/* an Observer */);
http://reactivex.io/documentation/operators/subscribeon.html
By default, RxJava is synchronous. It means that everything will be perform in the same thread (and the current thread), by default. You can perform tasks in another thread thanks to observeOn / subscribeOn methods, or using some operators that perform tasks in another job (because it use another scheduler, like delay, interval, ...)
In your example, you have to explitly set in which scheduler the subscription will pe performed. (here, in which thread Observable.from will emit your list)
test.getFromDb()
.subscribeOn(Schedulers.io())
.subscribe();
Then you can use the flatMap operator and calling your publishToQueue method. This method will be executed in the previous scheduler, but you can force it to use another scheduler, thanks to observeOn method. Everything after the observeOn method will be executed in another thread.
test.fromDb()
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.computation())
.flatMap(l -> test.publishToqueue(l))
.subscribe();
In the following piece of code, the observable is supposed to fire off every 300 milliseconds. I spiced it up by simulating background activity that takes 1 second. I was expecting that since I am using a scheduler that uses a thread pool underneath, the interval observable would keep firing on a new thread every 300 milliseconds. What happens instead, is that the interval observable waits for an entire second every time, before firing off again. Is this the desired behavior? How could one force it to fire off in parallel, if a task takes more than the requested amount of time?
Here is the code:
Observable
.interval(300, TimeUnit.MILLISECONDS, Schedulers.io())
.doOnNext(new Action1<Long>() {
#Override
public void call(Long aLong) {
System.out.println("action thread: " + Thread.currentThread());
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
})
.map(new Func1<Long, Float>() {
#Override
public Float call(Long aLong) {
final double result = Math.random();
return new Float(result);
}
})
.takeWhile(new Func1<Float, Boolean>() {
#Override
public Boolean call(Float aFloat) {
return aFloat >= 0.01f;
}
})
.subscribe(new Action1<Float>() {
#Override
public void call(Float aFloat) {
System.out.println("observing thread: " + Thread.currentThread());
System.out.println(aFloat);
}
});
Observables are sequential in nature so if you put in a sleep such as in example, you are blocking the whole sequence. To do background computation, you have to move it to another thread via observeOn or subscribeOn. In this case, you can flatMap/concatMapEager in another observable that does the sleeping and merge the results back into the main sequence:
Observable.interval(300, TimeUnit.MILLISECONDS)
.flatMap(t -> Observable.fromCallable(() -> {
Thread.sleep(1000);
}).subscribeOn(Schedulers.io()))
.map(...)
.takeWhile(...)
.subscribe(...)
I have just started learning RxJava and thinking reactive.
I have found some article (7 tips) that says subjects should only be used as a last resort or the first implementation, and I have been thinking..
As of now I have one PublishSubject
PublishSubject.create()
.scan(0, (a, b) -> a + 1).subscribe(replaySubject)
It is incrementing the value by 1 every time, and is subscribed by replaySubject
ReplaySubject.create();
That just prints the value. So at start I have 0, then on every PublishSubject.onNext I am incrementing the value, so I got 1, 2 etc. Because its replaySubject I am getting the whole chain.
However I have been thinking whether or not this may be done without subjects? I have been reading the RxJava Wiki and the operators, but I can't figure out any way this could be done.
Update
So the pseudo code I am trying to archieve is somehow to have one observable that starts with 1
Observable.just(1)
Now I do have on click listener
OnClick{ }
And every time I click on some button I want to see all previous numbers + 1, so 1 then 1, 2 next 1,2,3 etc.
I have been trying with ConnectableObservator.replay however this does not succeeded at all. And inside listener I jave been trying to first add scan on my Observable to increase value and then subscribe so I can print it. But this does not work either. Damn I think i'm in a black corner and misunderstood the idea of observables...
Since you are writing an Android app, you can use RxAndroid. Here is an example,
Observable.just(1).flatMap(new Func1<Integer, Observable<Integer>>() {
#Override
public Observable<Integer> call(Integer initialValue) {
return ViewObservable.clicks(button, false)
.subscribeOn(AndroidSchedulers.mainThread())
.scan(initialValue, new Func2<Integer, View, Integer>() {
#Override
public Integer call(Integer integer, View v) {
return integer + 1;
}
});
}
}).subscribe(new Observer<Integer>() {
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable e) {
e.printStackTrace();
}
#Override
public void onNext(Integer integer) {
System.out.println(integer);
}
});
I'm not sure if it's what you want. Maybe you only need:
ViewObservable.clicks(button, false)
.subscribeOn(AndroidSchedulers.mainThread())
.scan(1, new Func2<Integer, View, Integer>() {
#Override
public Integer call(Integer integer, View v) {
return integer + 1;
}
}).subscribe(new Observer<Integer>() {
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable e) {
e.printStackTrace();
}
#Override
public void onNext(Integer integer) {
System.out.println(integer);
}
});