I am learning RxJava and am testing a scenario where I read data from a DB and then post it to a Queue. I just made a sample mock of the whole process but I don't seem to find the Observable working as I wanted it to ie. asynchronously.
This is my code:
package rxJava;
import java.util.ArrayList;
import java.util.List;
import rx.Observable;
import rx.Observer;
import rx.functions.Action1;
public class TestClass {
public static void main(String[] args) {
TestClass test = new TestClass();
System.out.println("---START---");
test.getFromDB().subscribe(new Observer<String>() {
#Override
public void onCompleted() {
System.out.println("Publish complete.");
}
#Override
public void onError(Throwable t) {
System.out.println(t.getMessage());
}
#Override
public void onNext(String s) {
test.publishToQueue(s).subscribe(new Observer<Boolean>() {
#Override
public void onNext(Boolean b) {
if (b) {
System.out.println("Successfully published.");
}
}
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable arg0) {
}
});
};
});
System.out.println("---END---");
}
public Observable<String> getFromDB() {
List<String> list = new ArrayList<String>();
for (int i = 0; i < 30; i++) {
list.add(Integer.toString(i));
}
return Observable.from(list).doOnNext(new Action1<String>() {
#Override
public void call(String temp) {
if (temp.contains("2")) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
});
}
public Observable<Boolean> publishToQueue(String s) {
return Observable.defer(() -> {
try {
if (s.contains("7")) {
Thread.sleep(700);
}
System.out.println("Published:: " + s);
} catch (InterruptedException e) {
e.printStackTrace();
}
return Observable.just(true);
});
}
}
Suppose I get a list from the DB asynchronously and want to post it to the queue,. I have used an Observable returned from getFromDB and have subscribed to it which mimics the data I get from DB. Every time I get the data from DB , I want to push it to a queue using publishToQueue which also returns an Observable. I wanted to make the queue call also asynchronous. Now on positive acknowledgement from the queue such as the Boolean which I am returning (Observable<Boolean>), I want to print something.
So basically I just want both the processes to be asynchronous. For every data from DB, I push it to the Queue asynchronously.
I have added Thread.sleep() in both the methods, db call and queue so as to mimic a delay and to test the asynchronous operations. I think this is what causing the problem. But I also tried Obseravable.delay() but that doesn't even produce any output.
Please help me understand how this works and how I can make it work as I want it to.
You have to specified subscribeOn value.
Observable.just("one", "two", "three", "four", "five")
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(/* an Observer */);
http://reactivex.io/documentation/operators/subscribeon.html
By default, RxJava is synchronous. It means that everything will be perform in the same thread (and the current thread), by default. You can perform tasks in another thread thanks to observeOn / subscribeOn methods, or using some operators that perform tasks in another job (because it use another scheduler, like delay, interval, ...)
In your example, you have to explitly set in which scheduler the subscription will pe performed. (here, in which thread Observable.from will emit your list)
test.getFromDb()
.subscribeOn(Schedulers.io())
.subscribe();
Then you can use the flatMap operator and calling your publishToQueue method. This method will be executed in the previous scheduler, but you can force it to use another scheduler, thanks to observeOn method. Everything after the observeOn method will be executed in another thread.
test.fromDb()
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.computation())
.flatMap(l -> test.publishToqueue(l))
.subscribe();
Related
I have a DisposableSubscriber to a Flowable. The Flowable runs for some timeUntilTimeout, but in some situations I need to kill it earlier. Right now I call .dispose() on the DisposableSubscriber but the Flowable continues to emit events. Eventually the Flowable times out and .doOnCancel() is called.
I have the following code:
private Disposable mDisposableSubscription = null;
public void start() {
mDisposableSubscription = getFlowable()
.timeout(timeUntilTimeout, TimeUnit.MILLISECONDS)
.subscribeWith(new DisposableSubscriber<T>() {
#Override
public void onComplete() {
}
#Override
public void onError(Throwable throwable) {
dispose();
}
#Override
public void onNext(T t) {
// Do something
}
});
}
public void stop() {
// Log "dispose"
mDisposableSubscription.dispose();
}
private Flowable<T> getFlowable() {
return Flowable.create(new FlowableOnSubscribe<T>() {
public void subscribe(FlowableEmitter<T> emitter) {
startSomeAsyncProcess();
}
}).doOnCancel(new Action() {
public void run() {
// Log "do on cancel"
stopSomeAsyncProcess();
}
});
}
Calling stop() to dispose of the DisposableSubscriber before the Flowable times out means events emitted by the Flowable are no longer handled, but the Flowable continues emitting events and the async process continues running. I was under the impression that calling .dispose() downstream of the Flowable kills the Flowable by calling .doOnCancel(), but this does not appear to be the case. What am I missing?
The flowable is getting disposed, but you are not checking it on your Flowable.create function, so what happens is that the startSomeAsyncProcess() ignores it and keeps going.
To solve the issue, you should check the emitter.isDisposed() flag to know if you should stop emitting.
Example:
Flowable<T> getFlowable() {
return Flowable.create(new FlowableOnSubscribe<T>() {
public void subscribe(FlowableEmitter<T> emitter) {
while(!emitter.isDisposed()) {
emitter.onNext(...);
}
}
});
}
If that startSomeAsyncProcess() function doesn't allow you to check the flag, surely there is some way to cancel it. Then you can attach a cancellable:
Flowable<T> getFlowable() {
return Flowable.create(new FlowableOnSubscribe<T>() {
public void subscribe(FlowableEmitter<T> emitter) {
startSomeAsyncProcess();
emitter.setCancellable(() -> stopSomeAsyncProcess());
// I don't remember if it's setCancellable() or setDisposable()
}
});
}
Update: the methods setCancellable(...) and setDisposable(...) should behave equally, they just take different arguments.
I've got a data access object that passes each item in a data source to a consumer:
public interface Dao<T> {
void forEachItem(Consumer<T> item);
}
This always produces items in a single threaded way - I can't currently change this interface.
I wanted to create a Flowable from this interface:
private static Flowable<String> flowable(final Dao dao) {
return Flowable.create(emitter -> {
dao.forEachItem(item ->
emitter.onNext(item));
emitter.onComplete();
}, ERROR);
}
If I use this Flowable in a situation where the processing takes longer than the rate at which items are emitted then I understandably get a missing back pressure exception as I am using ERROR mode:
Dao<String> exampleDao =
itemConsumer ->
IntStream.range(0, 1_000).forEach(i ->
itemConsumer.accept(String.valueOf(i)));
flowable(exampleDao)
.map(v -> {
Thread.sleep(100);
return "id:" + v;
})
.blockingSubscribe(System.out::println);
I don't wish to buffer items - seems like this could lead to exhausting memory on very large data sets - if the operation is significantly slower than the producer.
I was hoping there would be a backpressure mode that would allow the emitter to block when passed next/completion events when it detects back pressure but that does not seem to be the case?
In my case as I know that the dao produces items in a single threaded way I thought I would be able to do something like:
dao.forEachItem(item -> {
while (emitter.requested() == 0) {
waitABit();
}
emitter.onNext(item)
});
but this seems to hang forever.
How wrong is my approach? :-) Is there a way of producing items in a way that respects downstream back pressure given my (relatively restrictive) set of circumstances?
I know I could do this with a separate process writing to a queue and then write a Flowable based on consuming from that queue- would that be the preferred approach instead?
Check the part of the Flowable, especially the part with Supscription.request(long). I hope that gets you on the right way.
The TestProducerfrom this example produces Integerobjects in a given range and pushes them to its Subscriber. It extends the Flowable<Integer> class. For a new subscriber, it creates a Subscription object whose request(long) method is used to create and publish the Integer values.
It is important for the Subscription that is passed to the subscriber that the request() method which calls onNext()on the subscriber can be recursively called from within this onNext() call. To prevent a stack overflow, the shown implementation uses the outStandingRequests counter and the isProducing flag.
class TestProducer extends Flowable<Integer> {
static final Logger logger = LoggerFactory.getLogger(TestProducer.class);
final int from, to;
public TestProducer(int from, int to) {
this.from = from;
this.to = to;
}
#Override
protected void subscribeActual(Subscriber<? super Integer> subscriber) {
subscriber.onSubscribe(new Subscription() {
/** the next value. */
public int next = from;
/** cancellation flag. */
private volatile boolean cancelled = false;
private volatile boolean isProducing = false;
private AtomicLong outStandingRequests = new AtomicLong(0);
#Override
public void request(long n) {
if (!cancelled) {
outStandingRequests.addAndGet(n);
// check if already fulfilling request to prevent call between request() an subscriber .onNext()
if (isProducing) {
return;
}
// start producing
isProducing = true;
while (outStandingRequests.get() > 0) {
if (next > to) {
logger.info("producer finished");
subscriber.onComplete();
break;
}
subscriber.onNext(next++);
outStandingRequests.decrementAndGet();
}
isProducing = false;
}
}
#Override
public void cancel() {
cancelled = true;
}
});
}
}
The Consumer in this example extends DefaultSubscriber<Integer> and on start and after consuming an Integer requests the next one. On consuming the Integer values, there is a little delay, so the backpressure will be built up for the producer.
class TestConsumer extends DefaultSubscriber<Integer> {
private static final Logger logger = LoggerFactory.getLogger(TestConsumer.class);
#Override
protected void onStart() {
request(1);
}
#Override
public void onNext(Integer i) {
logger.info("consuming {}", i);
if (0 == (i % 5)) {
try {
Thread.sleep(500);
} catch (InterruptedException ignored) {
// can be ignored, just used for pausing
}
}
request(1);
}
#Override
public void onError(Throwable throwable) {
logger.error("error received", throwable);
}
#Override
public void onComplete() {
logger.info("consumer finished");
}
}
in the following main method of a test class the producer and consumer are created and wired up:
public static void main(String[] args) {
try {
final TestProducer testProducer = new TestProducer(1, 1_000);
final TestConsumer testConsumer = new TestConsumer();
testProducer
.subscribeOn(Schedulers.computation())
.observeOn(Schedulers.single())
.blockingSubscribe(testConsumer);
} catch (Throwable t) {
t.printStackTrace();
}
}
When running the example, the logfile shows that the consumer runs continuously, while the producer only gets active when the internal Flowable buffer of rxjava2 needs to be refilled.
I have a ChatThreadsActivity() which displays messages between two people.
Whenever the user opens the activity, I make an API to fetch the most recent 100 messages. So in the onCreate() method I call makeApiRequestToGetChatThread().
I know I am doing alot of things wrong here.
Inside onNext() I add rows to the chatThreadAdapter. I know this is wrong since I update adapter after ever insert.
chatDaoObject.queryChatThreadsFromDB(someId).observeOn(AndroidSchedulers.mainThread()).subscribe()
Should this be anonymously called? When will this be unsubscribed if ever?
How do I unsubscribe from it?
I've read about BackPressure and realize this is it, I've run this on Android Monitor and here's how I can identify it. Am I doing this right?
private void makeApiRequestToGetChatThreads() {
public void onResponse(Call call, final Response response) {
final String responseString = response.body().string();
runOnUiThread (() -> {
final JSONArray array = new JSONArray(responseString);
JSONObject obj;
for (int i=0 ; i < array.length ; i++) {
obj = new JSONObject(array.get(i));
insertAChatIntoDB (obj);
}
}
});
}
private void insertAChatIntoDB(JSONObject o) {
if(insertSubscriber != null) {
insertSubscriber.unsubscribe();
}
insertSubscriber = new Subscriber<Long>() {
public void onCompleted() {
}
public void onError() {
}
public void onNext() {
chatDaoObject.queryChatThreadsFromDB(someId)
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Action1<List<ChatObject>>() {
#Override
public void call(List<ChatObject> chatList) {
if (chatList != null) {
//Royal Screw up
//After every insertion in the DB I update the UI
chatThreadAdapter.addAllChatThreadsIntoList(chatList);
//Notify is called inside the above below
//notifyItemRangeChanged(initialPosition,chatList.size())
}
}
}, new Action1<Throwable>() {
#Override
public void call(Throwable throwable) {
}
});
}
};
try {
//Inserts happening on Schedulers.computation()
chatDaoObject.insertAChatInDB(o).observeOn(Schedulers.computation()).subscribe(insertSub);
} catch (JSONException e) {
e.printStackTrace();
Crashlytics.logException(e);
}
}
I use RxJAVA and SQLBrite Dao and here's what the query looks like:
//In `ChatDao` class
ChatDataDao extends Dao {
...
...
public Observable<long> insertAChatInDB(JSONObject o) {
ChatThreadsTable.ContentValuesBuilder valuesBuilder = ChatThreadsTable.contentValues();
...
//populating columns values
...
return insert(ChatThreadsTable.TABLE_NAME, valuesBuilder.build(), SQLiteDatabase.CONFLICT_IGNORE);
}
public Observable<List> queryChatThreadsFromDB () {
return rawQuery("SELECT * FROM " + ChatThreadsTable.TABLE_NAME).run().mapToList(ChatObjectMapper.MAPPER);
}
...
...
}
Edit:
Is this the right way to query db without worrying about subscription/unsubscription?
rawQuery("SELECT * FROM " + SomeObject.TABLE_NAME + " ORDER BY " + SomeObject.COL1 + " , " +SomeObject.COL2 + " DESC").run().mapToList(SomeObjectMapper.MAPPER)
.flatMap(new Func1<List<SomeObject>, Observable<SomeObject>>() {
#Override
public Observable<SomeObject> call(List<SomeObject> SomeObjects) {
return Observable.from(SomeObjects);
}
}).doOnNext(new Action1<SomeObject>() {
#Override
public void call(SomeObject chatThreadObject) {
}
}).subscribe();
what is going on with all these tutorials telling people to create their own Subscribers? Here's a cleaned up version:
private void makeApiRequestToGetChatThreads() {
// ... some call that calls back onResponse
}
public void onResponse(Call call, final Response response) {
// are you sure this is how you parse a JSON String?
Observable
.from(response.body().string())
.observeOn(Schedulers.computation())
.flatMapIterable(JsonArray::new)
.map(JSONObject::new)
.flatMap(chatDaoObject::insertAChatInDB)
.flatMap(chatDaoObject::queryChatThreadsFromDB)
.observeOn(AndroidSchedulers.mainThread())
.doOnNext(chatThreadAdapter::addAllChatThreadsIntoList)
.subscribe(dummy -> {}, throwable -> {
e.printStackTrace();
Crashlytics.logException(e);
});
}
It's quite likely that there's a better way to directly make the network call return an Observable. Check your documentation / peers.
When the Observable completes all processing and UI updates, it will unsubscribe on it's own. However, if the network call takes a while to complete, the user might have already switched screens/apps, and the UI manipulation will break your app. Consider saving the subscription in your view and unsubscribing. Hint: creating the subscription in the onResponse is... not optimal.
You are getting 100s of threads, and updating the UI for each and every one of them. Are you sure you want that?
I don't think you need to care about backpressure.
So I have a list of Track Ids that for each track Id I need to execute a network request to get the track details, I am using a for loop to launch all the requests and a latch to wait for all the requests to be completed. After they are completed then the callback is sent with the List of Tracks that have already populated.
I would like to know if there is any better way to do this, maybe with RxJava ?
I am using Retrofit 2.0 in Android.
public IBaseRequest batchTracksById(final TrackIdList trackIdListPayload, final IRequestListener<TracksList> listener) {
final TracksList tracks = new TracksList();
final Track[] trackArray = newrack[trackIdListPayload.getTrackIds().length];
tracks.setTrack(trackArray);
final CountDownLatch latch = new CountDownLatch(trackArray.length);
Thread t = new Thread(new Runnable() {
#Override
public void run() {
try {
latch.await();
handler.post(new Runnable() {
#Override
public void run() {
listener.onRequestUpdate(null, tracks, null, true);
}
});
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
t.start();
for (String id : trackIdListPayload.getTrackIds()) {
getTrackById(id, new IRequestListener<Track>() {
#Override
public void onRequestFailure(IBaseRequest request, Exception exception) {
latch.countDown();
}
#Override
public void onRequestUpdate(IBaseRequest request, Track track, RequestState state, boolean requestComplete) {
//iterate through the tracks and update the ones in the thing
int i = 0;
for (String s : trackIdListPayload.getTrackIds()) {
if (s.equals(track.getTrackId())) {
trackArray[i] = track;
// don't break here, as we may have a case where we have multiple instances of the same trackId (although
// at the moment a request will be made for each anyway...
}
i++;
}
latch.countDown();
}
});
}
return null;
}
If you want to make all the requests asynchronously and wait for them to return you can do this (lambdas for brevity and readability):
tracks.flatMap(track -> getTrackDetails(track.id)
.subscribeOn(Schedulers.io()))
.toList()
.doOnNext(list -> processTrackList())
...
If you require that the results are returned in the order of tracks but are still requested asynchronously then in soon to be released rxjava 1.0.15 you will be able to do this
tracks.concatMapEager(track -> getTrackDetails(track.id)
.subscribeOn(Schedulers.io())
.toList()
.doOnNext(list -> processTrackList())
...
If I understand correctly, you have a list of tracks as input and you want a list of webservice results. Here's a simple way to do that with RxJava if you can make your network call synchronous (rxjava will handle the background processing for you).
Observable.from(trackList)
.map(new Func1<Track, Response>() {
#Override
public Response call(Track track) {
return makeRequestSynchronously(track.id());
}
})
.toList()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Subscriber<List<Response>>() {
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable e) {
}
#Override
public void onNext(List<Response> responses) {
}
});
Edit: You can change Retrofit to return observables from webservice, if you do that you will need to change map to the following
.flatMap(new Func1<Track, Observable<Response>>() {
#Override
public Observable<Response> call(Track track) {
return makeRequestThatReturnsObservable(track.id());
}
})
I am using PublishSubject in the class that is responsible for synchronization. When the synchronization is done all the subscribers will be notified. The same happens in case of an error.
I've noticed that the next time I subscribe after an error has occured, it is immediately return to the subscriber.
So the class may look like this:
public class Synchronizer {
private final PublishSubject<Result> mSyncHeadObservable = PublishSubject.create();
private final ThreadPoolExecutor mExecutor = new ThreadPoolExecutor(1, 1,
10, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>(true),
new ThreadPoolExecutor.DiscardPolicy());
public Observable<Result> syncHead(final int chunkSize) {
mExecutor.execute(new Runnable() {
#Override
public void run() {
try {
//Do some work which either returns a result or throws an error
//...
mSyncHeadObservable.onNext(Notification.createOnNext(/*some result*/));
} catch (Throwable error) {
mSyncHeadObservable.onError(Notification.<Result>createOnError(error));
}
}
});
Is there an observable which can just serve as an proxy? May be some other Rx approach?
UPDATE:
I've followed #akarnokd approach and emit the events wrapped into the RxJava Notification. Then unwrap them via flatMap(). So the clients of Synchronizer class won't need to do it.
//...
private PublishSubject<Notification<Result>> mSyncHeadObservable = PublishSubject.create();
public Observable<Result> syncHead(final int chunkSize) {
return mSyncHeadObservable.flatMap(new Func1<Notification<Result>, Observable<Result>>() {
#Override
public Observable<Result> call(Notification<Result> result) {
if (result.isOnError()) {
return Observable.error(result.getThrowable());
}
return Observable.just(result.getValue());
}
}).doOnSubscribe(
new Action0() {
#Override
public void call() {
startHeadSync(chunkSize);
}
});
}
private void startHeadSync(final int chunkSize) {
mExecutor.execute(new Runnable() {
#Override
public void run() {
try {
//Do some work which either returns a result or throws an error
//...
mSyncHeadObservable.onNext(Notification.createOnNext(/*some result*/));
} catch (Throwable error) {
mSyncHeadObservable.onError(Notification.<Result>createOnError(error));
}
}
});
}
//...
I'm not sure what your want to achieve with this setup, but generally, in order to avoid a terminal condition with PublishSubject, you should wrap your value and error into a common structure and always emit those, never any onError and onCompleted. One option is to use RxJava's own event wrapper, Notification, and your Subscribers should unwrap the value.
When a error occurred, the observable reached an terminal state.
If you want to continue to observe it, you should resubscribe to you observable with retry operator or use another error handling operators