We Keep Coding

How to test non-RxJava observables or async code in general?

I'm playing around with implementing my own observables or porting them from other languages for fun and profit.
The problem I've run into is that there's very little info on how to properly test observables or async code in general.
Consider the following test code:
// Create a stream of values emitted every 100 milliseconds
// `interval` uses Timer internally
final Stream<Number> stream =
Streams.interval(100).map(number -> number.intValue() * 10);
ArrayList<Number> expected = new ArrayList<>();
expected.add(0);
expected.add(10);
expected.add(20);
IObserver<Number> observer = new IObserver<Number>() {
public void next(Number x) {
assertEquals(x, expected.get(0));
expected.remove(0);
if(expected.size() == 0) {
stream.unsubscribe(this);
}
}
public void error(Exception e) {}
public void complete() {}
};
stream.subscribe(observer);
As soon as the stream is subscribed to, it emits the first value. onNext is called... And then the test exits successfully.
In JavaScript most test frameworks nowadays provide an optional Promise to the test case that you can call asynchronously on success/failure. Is anything similar available for Java?

Since the execution is asyncronious, you have to wait until is finish. You can just wait for some time in an old fashion way
your_code
wait(1000)
check results.
Or if you use Observables you can use TestSubscriber
In this example you can see how having an async operation we wait until the observer consume all items.
#Test
public void testObservableAsync() throws InterruptedException {
Subscription subscription = Observable.from(numbers)
.doOnNext(increaseTotalItemsEmitted())
.subscribeOn(Schedulers.newThread())
.subscribe(number -> System.out.println("Items emitted:" + total));
System.out.println("I finish before the observable finish. Items emitted:" + total);
new TestSubscriber((Observer) subscription)
.awaitTerminalEvent(100, TimeUnit.MILLISECONDS);
}
You can see more Asynchronous examples here https://github.com/politrons/reactive/blob/master/src/test/java/rx/observables/scheduler/ObservableAsynchronous.java

RxJava - Observers don't get onNext() items after a while

I have a system made up of several interconnected components. Everything works well for a while, but after a certain amount of time, a couple of observers stop receiving the items sent by an observable's onNext().
A simplified scenario is like this: I have
Component1.start() -> creates a ConnectableObservable with Observable.create(...).subscribeOn().observeOn().publish(), and subscribes Component2. After that, it connect()s. This observable emits some items in a loop and then invokes s.onComplete() when it finishes.
Component2 implements Observer. In addition it has a ConnectableObservable that runs a while(true) loop. When it gets a value in its onNext(), called by Component1, it notifies Component0 using its own ConnectableObservable. (NOTE I also implemented them with PublishSubject and the same happens).
Component1.start() //Creates Component1's ConnectableObservable, subscribes Component2 and starts running with connect();
Component1.connectableObservable -> onNext() ---> Component2
Component2.connectableObservable -> onNext() ---> Component0
When Component0.onNext() gets a particular item (after 100 iterations), it stops Component1.observable, making it exit its loop and call onComplete().
After some time, Component0 calls Component1.start() and everything starts again.
What I've seen is that, when everything is ok Component1.observable.onNext() calls rx.internal.operators.OperatorSubscribeOn.......subscriber.onNext()
rx.internal.operators.OperatorSubscribeOn
#Override
public void call(final Subscriber<? super T> subscriber) {
final Worker inner = scheduler.createWorker();
subscriber.add(inner);
inner.schedule(new Action0() {
#Override
public void call() {
final Thread t = Thread.currentThread();
Subscriber<T> s = new Subscriber<T>(subscriber) {
#Override
public void onNext(T t) {
subscriber.onNext(t);
subscriber.onNext() is the inner class private static final class ObserveOnSubscriber<T> and here ends up invoking schedule():
#Override
public void onNext(final T t) {
if (isUnsubscribed() || finished) {
return;
}
if (!queue.offer(on.next(t))) {
onError(new MissingBackpressureException());
return;
}
schedule();
}
schedule() is
protected void schedule() {
if (counter.getAndIncrement() == 0) {
recursiveScheduler.schedule(this);
}
}
counter is 0 so recursiveScheduler.schedule(this); is invoked and Component2 gets the item.
Now, when it stops working what happens is that counter isn't 0 anymore, actually every call increments it. Thus, recursiveScheduler.schedule(this); is never called and Component2 doesn't get anything.
What could be the cause of this? Why is counter 0 and at some point starts increasing?
UPDATE: Digging in the source code I've seen the following: after schedule() is called, there's a scheduled task that invokes the code below, decreasing counter when the items haven't been missed:
private static final class ObserveOnSubscriber<T> extends Subscriber<T> implements Action0 {
// only execute this from schedule()
#Override
public void call() {
...
emitted = currentEmission;
missed = counter.addAndGet(-missed);
if (missed == 0L) {
break;
}
According to this, since the items are missed, counter increases and then subsequent items are also missed.
What could be the reason why items are missed?
I've noticed something weird. If I remove any of the other (not mentioned) observables from the program, no items are missed ever. They have Component0 as observer and produce their items in their own subscribeOn() thread so I can't see how they affect this scenario.
UPDATE 2: I've kept trying to find out what happens. When I do Component1.connectableObservable.connect(), it ends up calling private static final class ObserveOnSubscriber<T> extends Subscriber<T> implements Action0 -> init()
Here schedule() is called:
void init() {
// don't want this code in the constructor because `this` can escape through the
// setProducer call
Subscriber<? super T> localChild = child;
localChild.setProducer(new Producer() {
#Override
public void request(long n) {
if (n > 0L) {
BackpressureUtils.getAndAddRequest(requested, n);
schedule();
The right behaviour leaves OperatorObserveOn.counter = 0 after schedule(). When it doesn't work anymore, scheduler() increases +1 the value of OperatorObserveOn.counter.

Get the observer for every subscription and create a listener to tell you when one observer unsubscribe from the observable, then you can understand why this is happening.
In your case anyway I would take a look to Relay, since you dont have to unsubscribe your observable it´s more secure, and you can be sure it will never stop emitting events.
Take a look of this example.
https://github.com/politrons/reactive/blob/master/src/test/java/rx/relay/Relay.java

Subscribe based on other emissions

I have an observable that emits values. Based on these values I need to subscribe/unsubscribe to/from another Observable.
Is there a handy way of doing so? A convenient way instead creating a field for the subscription and handling it manually?
Example:
Observable A emits Booleans. If it emits true then a subscription should be made to Observable B - if false this subscription should be unsubscribed.

I'm not sure if we're 100% on the same page but I think you're missing one point. Maybe you'll think I'm nitpicking, but I think it will be good to get our terms straight.
Observable starts emitting values when a Subscriber subscribes to it. So unless you're thinking about two separate Subscribers you can't react to an emitted value with a subscription because the Observer won't emit anything.
That said... what (I think) you wanna do could be done this way:
Observable<Boolean> observableA = /* observable A initialization */;
final Observable<SomeObject> observableB = /* observable B initialization */;
observableA
.flatMap(new Func1<Boolean, Observable<SomeObject>>() {
#Override
public Observable<SomeObject> call(Boolean aBoolean) {
if (!aBoolean) {
throw new IllegalStateException("A dummy exception that is here just to cause the subscription to finish with error.");
}
return observableB;
}
})
.subscribe(
new Action1<SomeObject>() {
#Override
public void call(SomeObject someObject) {
// THIS IS A PART OF THE SUBSCRIBER TO OBSERVABLE B.
// THIS METHOD WILL BE CALLED ONLY IF THE OBSERVABLE A RETURNED TRUE
}
},
new Action1<Throwable>() {
#Override
public void call(Throwable throwable) {
// A dummy Action1 so the subscription does not crash on the Exception
}
});

If all of observables has the same type or you can combine whatever you want based on values.
Observable.from(new int[]{1,2,3,4,5})
.filter(i -> i < 5) // filter out something
.flatMap(i -> {
if (i < 2) { // subscribe on some observable, based on item value
return Observable.just(i);
} else {
return Observable.just(3);
}
})

Observable.publish() doesn't call onCompleted() on observers that subscribe after the source Observable is done

I'm trying to get an Observable to share its emissions with all the subscribers, so that it would be subscribe()d to exactly once.
I tried using Observable.publish(), but it appears that subscribers to the published Observable don't receive any termination messages( onCompleted() and possibly onError()) if they subscribe after the source Observable is done. Here is a piece of code to demonstrate that:
static <T> Observer<T> printObserver(String name) {
return new Observer<T>() {
#Override public void onCompleted() {
System.out.println(name + ": onCompleted()");
}
#Override public void onError(Throwable e) {
System.out.println(name + ": onError( " + e + " )");
}
#Override public void onNext(T value) {
System.out.println(name + ": onNext( " + value + " )");
}
};
}
public void testRxPublishConnect() throws Exception {
Observable<Integer> sourceObservable = Observable.range(1, 5);
ConnectableObservable<Integer> sharedObservable = sourceObservable.publish();
sharedObservable.subscribe(printObserver("Observer #1"));
sharedObservable.connect();
sharedObservable.subscribe(printObserver("Observer #2"));
}
This is what gets printed:
Observer #1: onNext( 1 )
Observer #1: onNext( 2 )
Observer #1: onNext( 3 )
Observer #1: onNext( 4 )
Observer #1: onNext( 5 )
Observer #1: onCompleted()
Note that Observer #2 doesn't receive onCompleted().
I don't think this is the desired behavior. Am I missing something?
I tried it in RxJava versions 1.0.8 and 1.0.14 with the same result.

Try .share() which is .publish().refCount().

This is by design. If you call connect() in this case, your subscriber will receive all events from the start. If a terminated publish would terminate its child subscribers immediately, you likely couldn't observe values because once connected, publish ticks away its source slowly if there are no subscribers to it.

I'm 99% sure this is the expected behavior. I'm not sure about RxJava, but in most of the implementations of the publish&subscribe pattern that I know of, the default behavior for an observable is to publish events to subscribers and forget about them. This means that notifications are not 'retro-active' (i.e. subscribers don't get to know anything about the events emitted in the past).
Also, from the Observable Contract (section 'multiple observers') of the RxJava documentation :
If a second observer subscribes to an Observable that is already emitting items to a first observer, it is up to the Observable whether it will thenceforth emit the same items to each observer ... There is no general guarantee that two observers of the same Observable will see the same sequence of items.

Publish works by building a list of all subscribers then once connect() is called it starts producing data to all subscribers in it's subscriber list. This means all the subscribers have to be known before calling connect. Here's how you would use publish() or possibly more preferable the publish(Func1<Observable<T>, Observable<R>>) overload.
Known number of subscribers: Publish
Func closing over all subscriptions.
observableStream.publish(new Func1<Observable<Integer>, Observable<Integer>>() {
#Override
public Observable<Integer> call(Observable<Integer> subject) {
Observable<Integer> o1 = subject.doOnNext(somework1());
Observable<Integer> o2 = subject.doOnNext(somework2());
return Observable.merge(o1, o2);
}
});
Manual call to connect and subscribe:
ConnectableObservable<Integer> subject = observableStream.publish();
subject.subscribe(somework1());
subject.subscribe(somework2());
subject.connect();
If you don't know how many subscribers you'll have then you can window the inputs to manageable chunks and then publish your inputs over your collection of Transformers.
Unknown number of subscribers: Window
final Set<Transformer<Integer, String>> transformers = new HashSet<>();
observableStream
.window(100, TimeUnit.MILLISECONDS, 1000)
.flatMap(new Func1<Observable<Integer>, Observable<String>>(){
#Override
public Observable<String> call(Observable<Integer> window) {
return window.publish(new Func1<Observable<Integer>, Observable<String>>() {
#Override
public Observable<String> call(Observable<Integer> publish) {
Observable<Observable<String>> workObservables = Observable.from(transformers)
.map(new Func1<Transformer<Integer, String>, Observable<String>>(){
#Override
public Observable<String> call(Transformer<Integer, String> transformer) {
return publish.compose(transformer);
}});
return Observable.merge(workObservables);
}});
}})
.subscribe();
There is a third option. You could use observable.cache() but this will hold all input data from that observable stream in memory so you want to be careful with how you use that. In that case you'll probably end up windowing anyway to control the bounds of your cached subject.

Queue with notifications on isEmpty() changes

I have an BlockingQueue<Runnable>(taken from ScheduledThreadPoolExecutor) in producer-consumer environment. There is one thread adding tasks to the queue, and a thread pool executing them.
I need notifications on two events:
First item added to empty queue
Last item removed from queue
Notification = writing a message to database.
Is there any sensible way to implement that?

A simple and naïve approach would be to decorate your BlockingQueue with an implementation that simply checks the underlying queue and then posts a task to do the notification.
NotifyingQueue<T> extends ForwardingBlockingQueue<T> implements BlockingQueue<T> {
private final Notifier notifier; // injected not null
…
#Override public void put(T element) {
if (getDelegate().isEmpty()) {
notifier.notEmptyAnymore();
}
super.put(element);
}
#Override public T poll() {
final T result = super.poll();
if ((result != null) && getDelegate().isEmpty())
notifier.nowEmpty();
}
… etc
}
This approach though has a couple of problems. While the empty -> notEmpty is pretty straightforward – particularly for a single producer case, it would be easy for two consumers to run concurrently and both see the queue go from non-empty -> empty.
If though, all you want is to be notified that the queue became empty at some time, then this will be enough as long as your notifier is your state machine, tracking emptiness and non-emptiness and notifying when it changes from one to the other:
AtomicStateNotifier implements Notifier {
private final AtomicBoolean empty = new AtomicBoolean(true); // assume it starts empty
private final Notifier delegate; // injected not null
public void notEmptyAnymore() {
if (empty.get() && empty.compareAndSet(true, false))
delegate.notEmptyAnymore();
}
public void nowEmpty() {
if (!empty.get() && empty.compareAndSet(false, true))
delegate.nowEmpty();
}
}
This is now a thread-safe guard around an actual Notifier implementation that perhaps posts tasks to an Executor to asynchronously write the events to the database.

The design is most likely flawed but you can do it relatively simple:
You have a single thread adding, so you can check before adding. i.e. pool.getQueue().isEmpty() - w/ one producer, this is safe.
Last item removed cannot be guaranteed but you can override beforeExecute and check the queue again. Possibly w/ a small timeout after isEmpty() returns true. Probably the code below will be better off executed in afterExecute instead.
protected void beforeExecute(Thread t, Runnable r) {
if (getQueue().isEmpty()){
try{
Runnable r = getQueue().poll(200, TimeUnit.MILLISECONDS);
if (r!=null){
execute(r);
} else{
//last message - or on after execute by Setting a threadLocal and check it there
//alternatively you may need to do so ONLY in after execute, depending on your needs
}
}catch(InterruptedException _ie){
Thread.currentThread().interrupt();
}
}
}
sometime like that
I can explain why doing notifications w/ the queue itself won't work well: imagine you add a task to be executed by the pool, the task is scheduled immediately, the queue is empty again and you will need notification.