I'm getting to know Java 8 Stream API and I am unsure how to signal to a consumer of a stream that the stream is completed.
In my case the results of the stream-pipeline will be written in batches to a database or published on a messaging service. In such cases the stream-pipeline should invoke a method to "flush" and "close" the endpoints once the stream is closed.
I had a bit of exposure to the Observable pattern as implemented in RxJava and remember the Observer#onComplete method is used there for this purpose.
On the other hand Java 8 Consumer only exposes an accept method but no way to "close" it. Digging in the library I found a sub-interface of Consumer called Sink which offers an end method, but it's not public. Finally I thought of implementing a Collector which seems to be the most flexible consumer of a stream, but isn't there any simpler option?
The simplest way of doing a final operation is by placing the appropriate statement right after the terminal operation of the stream, for example:
IntStream.range(0, 100).parallel().forEach(System.out::println);
System.out.println("done");
This operation will be performed in the successful case only, where a commit is appropriate. While the Consumers run concurrently, in unspecified order, it is still guaranteed that all of them have done their work upon normal return.
Defining an operation that is also performed in the exceptional case is not that easy. Have a look at the following example:
try(IntStream is=IntStream.range(0, 100).onClose(()->System.out.println("done"))) {
is.parallel().forEach(System.out::println);
}
This works like the first one but if you test it with an exceptional case, e.g.
try(IntStream is=IntStream.range(0, 100).onClose(()->System.out.println("done"))) {
is.parallel().forEach(x -> {
System.out.println(x);
if(Math.random()>0.7) throw new RuntimeException();
});
}
you might encounter printouts of numbers after done. This applies to all kind of cleanup in the exceptional case. When you catch the exception or process a finally block, there might be still running asynchronous operations. While it is no problem rolling back a transaction in the exceptional case at this point as the data is incomplete anyway, you have to be prepared for still running attempts to write items to the now-rolled-back resource.
Note that Collector-based solutions, which you thought about, can only define a completion action for the successful completion. So these are equivalent to the first example; just placing the completing statement after the terminal operation is the simpler alternative to the Collector.
If you want to define operations which implement both, the item processing and the clean up steps, you may create your own interface for it and encapsulate the necessary Stream setup into a helper method. Here is how it might look like:
Operation interface:
interface IoOperation<T> {
void accept(T item) throws IOException;
/** Called after successfull completion of <em>all</em> items */
default void commit() throws IOException {}
/**
* Called on failure, for parallel streams it must set the consume()
* method into a silent state or handle concurrent invocations in
* some other way
*/
default void rollback() throws IOException {}
}
Helper method implementation:
public static <T> void processAllAtems(Stream<T> s, IoOperation<? super T> c)
throws IOException {
Consumer<IoOperation> rollback=io(IoOperation::rollback);
AtomicBoolean success=new AtomicBoolean();
try(Stream<T> s0=s.onClose(() -> { if(!success.get()) rollback.accept(c); })) {
s0.forEach(io(c));
c.commit();
success.set(true);
}
catch(UncheckedIOException ex) { throw ex.getCause(); }
}
private static <T> Consumer<T> io(IoOperation<T> c) {
return item -> {
try { c.accept(item); }
catch (IOException ex) { throw new UncheckedIOException(ex); }
};
}
Using it without error handling might be as easy as
class PrintNumbers implements IoOperation<Integer> {
public void accept(Integer i) {
System.out.println(i);
}
#Override
public void commit() {
System.out.println("done.");
}
}
processAllAtems(IntStream.range(0, 100).parallel().boxed(), new PrintNumbers());
Dealing with errors is possible, but as said, you have to handle the concurrency here. The following example does also just print number but use a new output stream that should be closed at the end, therefore the concurrent accept calls have to deal with concurrently closed streams in the exceptional case.
class WriteNumbers implements IoOperation<Integer> {
private Writer target;
WriteNumbers(Writer writer) {
target=writer;
}
public void accept(Integer i) throws IOException {
try {
final Writer writer = target;
if(writer!=null) writer.append(i+"\n");
//if(Math.random()>0.9) throw new IOException("test trigger");
} catch (IOException ex) {
if(target!=null) throw ex;
}
}
#Override
public void commit() throws IOException {
target.append("done.\n").close();
}
#Override
public void rollback() throws IOException {
System.err.print("rollback");
Writer writer = target;
target=null;
writer.close();
}
}
FileOutputStream fos = new FileOutputStream(FileDescriptor.out);
FileChannel fch = fos.getChannel();
Writer closableStdIO=new OutputStreamWriter(fos);
try {
processAllAtems(IntStream.range(0, 100).parallel().boxed(),
new WriteNumbers(closableStdIO));
} finally {
if(fch.isOpen()) throw new AssertionError();
}
Terminal operations on Java 8 streams (like collect(), forEach() etc) will always complete the stream.
If you have something that is processing objects from the Stream you know when the stream ends when the Collector returns.
If you just have to close your processor, you can wrap it in a try-with-resource and perform the terminal operatoni inside the try block
try(BatchWriter writer = new ....){
MyStream.forEach( o-> writer.write(o));
}//autoclose writer
You can use Stream#onClose(Runnable) to specify a callback to invoke when the stream is closed. Streams are pull-based (contrary to push-based rx.Observable), so the hook is associated with stream, not it's consumers
This is a bit of a hack but works well.
Create a stream concatenation of the original + a unique object.
Using peek(), see if the new object is encountered, and call the onFinish action.
Return the stream with filter, so that the unique object won't be returned.
This preserves the onClose event of the original stream.
public static <T> Stream<T> onFinish(Stream<T> stream, Runnable action) {
final Object end = new Object(); // unique object
Stream<Object> withEnd = Stream.concat(stream.sequential(), Stream.of(end));
Stream<Object> withEndAction = withEnd.peek(item -> {
if (item == end) {
action.run();
}
});
Stream<Object> withoutEnd = withEndAction.filter(item -> item != end);
return (Stream<T>) withoutEnd;
}
Another option is to wrap the original spliterator, and when it returns false, call the action.
public static <T> Stream<T> onFinishWithSpliterator(Stream<T> source, Runnable onFinishAction) {
Spliterator<T> spliterator = source.spliterator();
Spliterator<T> result = new Spliterators.AbstractSpliterator<T>(source.estimateSize(), source.characteristics()) {
#Override
public boolean tryAdvance(Consumer<? super T> action) {
boolean didAdvance = source.tryAdvance(action);
if (!didAdvance) {
onFinishAction.run();
}
return didAdvance;
}
};
// wrap the the new spliterator with a stream and keep the onClose event
return StreamSupport.stream(result, false).onClose(source::close);
}
Related
I want to create an Observable from view click listener using RxJava 2. I started from the simplest implementation (I don't use lambdas here to show you different types in this method):
Observable<View> viewObservable = Observable.create(new ObservableOnSubscribe<View>() {
#Override
public void subscribe(#NonNull ObservableEmitter<View> e) throws Exception {
mNewWordView.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View value) {
if (!e.isDisposed()) {
e.onNext(value);
}
}
});
}
});
Then I thought about the way to set onClickListener to null if it is not needed further. I found that there are two methods with similar (as for me) names:
e.setCancellable(Cancellable c); and
e.setDisposable(Disposable d);
What is the difference between them and which should I use?
From the Javadoc:
[Cancellable is] A functional interface that has a single cancel method that can throw.
The Disposable is not a functional interface plus when implementing its dispose() method, you are not allowed to throw checked exceptions.
In contrast, many non-RxJava components return a Closeable or AutoCloseable which are defined via throws IOException and throws Exception and are somewhat of a burden because you'd need try-catch it.
For example, you'd want to use setCancellable when you work with a file:
Observable.create((ObservableEmitter<byte[]> e) -> {
FileInputStream fin = new FileInputStream("raw.dat");
e.setCancellable(fin::close);
byte[] buffer = new byte[4096];
for (;;) {
int r = fin.read(buffer);
if (r < 0) {
break;
}
e.onNext(buffer);
}
e.onComplete();
});
and you'd use setDisposable if you use a Scheduler:
Observable.create((ObservableEmitter<Event> e) -> {
Worker worker = Schedulers.io().createWorker();
e.setDisposable(worker);
eventSource.onEvent(es ->
worker.schedule(() -> e.onNext(es))
);
});
The Cancellable ends up getting wrapped into a Disposable so the final effect is the same for either call. The difference is that Disposable has more features that you might not want to implement, so the simple Cancellable interface is there as an alternative.
If you just want to dispose something when the observable ends use Cancellable. If you have a resource that might be disposed because of some external reason you want to implement Disposable to implement the Disposable.isDisposed() method.
Do note that the methods are mutually exclusive. Only a single disposable or cancellable can be registered at once. Calling both overwrites the first one.
I am using Mockito to mock some objects and test my WebSocket message sender service. The send method takes a org.springframework.web.socket.WebSocketSession and a message, and returns a CompletableFuture.
In the lambda passed to the thenAccept() method of the CompletableFuture, I verify that the session.sendMessage() method has been called with the expected value :
WebSocketSession session = mockWebSocketSession();
TextMessage expectedMessage = new TextMessage("test text message");
sender.sendStringMessage(session, "test text message").thenAccept(nil -> {
try{ // this is what I am talking about
verify(session).sendMessage(expectedMessage);
}catch(IOException e){}
});
Since the sendMessage() method throws an IOException I am forced to add a useless try/catch block around the call on the inside the lambda. It is needlessly verbose.
How could I avoid it ?
You can try using Durian library
foodOnPlate.forEach(Errors.suppress().wrap(this::eat));
list.forEach(Errors.rethrow().wrap(c -> somethingThatThrows(c)));
or extend Cosumer yorself
#FunctionalInterface
public interface ThrowingConsumer<T> extends Consumer<T> {
#Override
default void accept(final T elem) {
try {
acceptThrows(elem);
} catch (final Exception e) {
/* Do whatever here ... */
System.out.println("handling an exception...");
throw new RuntimeException(e);
}
}
void acceptThrows(T elem) throws Exception;
}
//and then pass
thenAccept((ThrowingConsumer<String>) aps -> {
// maybe some other code here...
throw new Exception("asda");
})
I would rework you test in this way
final String testMessage = "test text message";
WebSocketSession session = mockWebSocketSession();
sender.sendStringMessage(session, testMessage).get(); // will wait here till operation completion
verify(session).sendMessage(new TextMessage(testMessage));
and add IOException to the test method signature.
This solution solves 2 issues:
you test code is cleaner and all you assertions and verifications are at the end of your test method in one place;
the solution solves race condition when you test may finish silently and green but your vitrification in CompletableFuture lambda even been executed
Follow on from my comment, I'll do something like this :
public void myMethod() {
try{ // this is what I am talking about
verify(session).sendMessage(expectedMessage);
}catch(IOException e) {}
}
And then :
sender.sendStringMessage(session, "test text message").thenAccept(nil -> myMethod());
duplicate of this ?
Java 8: Mandatory checked exceptions handling in lambda expressions. Why mandatory, not optional?
you are in test class, so just add throws IOException to your method. that way if this method raised IOException in which case it means your test will fail.
or you can also say that your method is expected to throw IOException,
something like :
#Test(expected = IOException.class)
public void yourTestCaser(){
//...
with that , it could look like this :
sender.sendStringMessage(session, "test text message").thenAccept(nil ->
{ verify(session).sendMessage(expectedMessage); });
I'm working on a project that involves Hystrix, and I decided to use RxJava. Now, forget Hystrix for the rest of this because I believe the main problem is with my complete screwing up of writing the Observable code correctly.
Need:
I need a way to return an observable that represents a number of observables, each running a user task. I want that Observable to be able to return all results from the tasks, even errors.
Problem:
Observable streams die on errors. If I have three tasks and the second task throws an exception, I never receive the third task even if it would have succeeded.
My Code:
public <T> Observable<T> observeManagedAsync(String groupName,List<EspTask<T>> tasks) {
return Observable
.from(tasks)
.flatMap(task -> {
try {
return new MyCommand(task.getTaskId(),groupName,task).toObservable().subscribeOn(this.schedulerFactory.get(groupName));
} catch(Exception ex) {
return Observable.error(ex);
}
});
}
Given that MyCommand is a class that extends HystrixObservableCommand, it returns an Observable and so shouldn't figure in on the problems I'm seeing.
Attempt 1:
Used Observable.flatMap as above
Good: Each Command is scheduled on it's own thread and the tasks run asynchronously.
Bad: On first Command exception, Observable completes having emitted previous successful results and emitting the Exception. Any in-flight Commands are ignored.
Attempt 2:
Used Observable.concatMapDelayError instead of flatMap
Bad: For some reason, tasks run synchronously. Why??
Good: I get all the successful results.
~Good: OnError gets a Composite exception with a list of the exceptions thrown.
Any help will be greatly appreciated and probably result in me being very embarrassed for not having thought of it myself.
Additional Code
This test succeeds with Observable.flatMap, but fails when using Observable.concatMapDelayError because the tasks do not run asynchronously:
java.lang.AssertionError: Execution time ran over the 350ms limit: 608
#Test
public void shouldRunManagedAsyncTasksConcurrently() throws Exception {
Observable<String> testObserver = executor.observeManagedAsync("asyncThreadPool",getTimedTasks());
TestSubscriber<String> testSubscriber = new TestSubscriber<>();
long startTime = System.currentTimeMillis();
testObserver.doOnError(throwable -> {
System.out.println("error: " + throwable.getMessage());
}).subscribe(testSubscriber);
System.out.println("Test execution time: "+(System.currentTimeMillis()-startTime));
testSubscriber.awaitTerminalEvent();
long execTime = (System.currentTimeMillis()-startTime);
System.out.println("Test execution time: "+execTime);
testSubscriber.assertCompleted();
System.out.println("Errors: "+testSubscriber.getOnErrorEvents());
System.out.println("Results: "+testSubscriber.getOnNextEvents());
testSubscriber.assertNoErrors();
assertTrue("Execution time ran under the 300ms limit: "+execTime,execTime>=300);
assertTrue("Execution time ran over the 350ms limit: "+execTime,execTime<=350);
testSubscriber.assertValueCount(3);
assertThat(testSubscriber.getOnNextEvents(),containsInAnyOrder("hello","wait","world"));
verify(this.mockSchedulerFactory, times(3)).get("asyncThreadPool");
}
Tasks for the above unit test:
protected List<EspTask<String>> getTimedTasks() {
EspTask longTask = new EspTask("helloTask") {
#Override
public Object doCall() throws Exception {
Thread.currentThread().sleep(100);
return "hello";
}
};
EspTask longerTask = new EspTask("waitTask") {
#Override
public Object doCall() throws Exception {
Thread.currentThread().sleep(150);
return "wait";
}
};
EspTask longestTask = new EspTask("worldTask") {
#Override
public Object doCall() throws Exception {
Thread.currentThread().sleep(300);
return "world";
}
};
return Arrays.asList(longTask, longerTask, longestTask);
}
You can use Observable.onErrorReturn(), and return special value (e.g. null), then filter non-special values downstream. Keep in mind that source observable will complete on error. Also depending on use case Observable.onErrorResumeNext()methods can be useful aswell. If you are interested in error notifications, use Observable.materialize(), this will convert items and onError(), onComplete() into Notifications, which then can be filtered by Notification.getKind()
Edit.
All operators mentioned above should be added right after .toObservable().subscribeOn(this.schedulerFactory.get(groupName)); assuming try/catch was absent.
You want to use mergeDelayError:
public <T> Observable<T> observeManagedAsync(String groupName,List<EspTask<T>> tasks) {
return Observable.mergeDelayError(Observable
.from(tasks)
.map(task -> {
try {
return new MyCommand(task.getTaskId(),groupName,task).toObservable().subscribeOn(this.schedulerFactory.get(groupName));
} catch(Exception ex) {
return Observable.error(ex);
}
}));
}
Note that your MyCommand constructor should not throw any exceptions; this allows your code to be written more concisely:
public <T> Observable<T> observeManagedAsync(String groupName,List<EspTask<T>> tasks) {
return from(tasks)
.map(task -> new MyCommand(task.getTaskId(), groupName, task)
.toObservable()
.subscribeOn(this.schedulerFactory.get(groupName)))
.compose(Observable::mergeDelayError);
}
Keep in mind that this will still invoke onError at most once; if you need explicit handling of all errors, use something like an Either<CommandResult, Throwable> as the return type (or handle the errors and return an empty observable).
Use .materialize() to allow all emissions and errors to come through as wrapped notifications then deal with them as you wish:
.flatMap(task -> {
try {
return new MyCommand(task.getTaskId(),groupName,task)
.toObservable()
.subscribeOn(this.schedulerFactory.get(groupName))
.materialize();
} catch(Exception ex) {
return Observable.error(ex).materialize();
}
});
Callable throws Exception, Runnable doesn't.
Is there anything standard that looks like
#FunctionalInterface
public interface TypedBlock<E extends Exception> {
public void run() throws E;
}
No, there is no built in functionality as I know. But you can use an external library for that (and many other cool features).
You can either use JOOL, where you can use the Unchecked class for this.
The example from there page demonstrates this with an IOException
Arrays.stream(dir.listFiles()).forEach(
Unchecked.consumer(file -> { System.out.println(file.getCanonicalPath()); })
);
Another (and in my opinion better) approach would be to use a functional designed library like Functionaljava.
A good approach would be to wrap your task in a Validation to decide afterwards, if the result was successful. This could look like this:
TypedBlock<IOException> foo = ...;
// do your work
final Validation<IOException, Unit> validation = Try.f(() -> {
foo.run();
return Unit.unit(); // Unit equals nothing in functional languages
})._1();
// check if we got a failure
if (validation.isFail()) {
System.err.println("Got err " + validation.fail());
}
// check for success
if (validation.isSuccess()) {
System.out.println("All was good :-)");
}
// this will just print out a message if we got no error
validation.forEach(unit -> System.out.println("All was good"));
There is java.lang.AutoCloseable which has a ()->{} throws Exception signature, however, it is burden with a predefined semantic. So for an ad-hoc use it might be suitable but when you design an API, I recommend defining your own interface.
Note that your specialized interface could still extend Callable<Void> to be a standard interface:
interface Block<E extends Exception> extends Callable<Void>{
void run() throws E;
#Override default Void call() throws E { run(); return null; }
/** This little helper method avoids type casts when a Callable is expected */
static <T extends Exception> Block<T> make(Block<T> b) { return b; }
}
This way you can use your Block interface with existing APIs:
// Example
ExecutorService e=Executors.newSingleThreadExecutor();
try {
e.submit(Block.make(()->{ throw new IOException("test"); })).get();
} catch (InterruptedException ex) {
throw new AssertionError(ex);
} catch (ExecutionException ex) {
System.out.println("received \""+ex.getCause().getMessage()+'"');
}
e.shutdown();
Note the trick with the static method Block.make. Without it you would have to cast the lambda expression to (Block<IOException>) instead of profiting from the improved type inference. This is only necessary where a Callable is expected, for your own API where a Block is expected, you can use lambda expressions and method references directly.
AFAIK submitting Callable/Runnable to ExecutorService is the way to go if I want to execute resource-heavy code in parallel. Hence my method structure:
public class ServiceClass {
protected final ExecutorService executorService = Executors.newCachedThreadPool();
public Future<Result> getResult(Object params) {
if (params == null) {
return null; // In situations like this the method should fail
}
// Do other fast pre-processing stuff
return executorService.submit(new CallProcessResult(params));
}
private class CallProcessResult implements Callable<Result> {
private Object params;
public CallProcessResult(Object params) {
this.params = params;
}
#Override
public Result call() throws Exception {
// Compute result for given params
// Failure may happen here too!
return result;
}
}
}
public class Result {
...
}
I have marked 2 spots in the code above in which failures can happen. The options available for error handling are quite different for those 2 cases.
Before submitting the task there can be issues like invalid parameters, some fast pre-processing code that may fail.
I see several ways to signify failure here:
In case of invalid params supplied to getResult return null immediately. In this case I'll have to check if getResult returned null every time I call it.
Throw checked exceptions instead of the above.
Instantiate a Future<Result> that returns null on get() request. I would do that with Apache Commons ConcurrentUtils.constantFuture(null). In this case I would expect getResult to always return some non-null Future<Result>. I like this option more, because it is consistent with the second case.
During task execution I can expect serious errors like lack of memory, corrupted files, unavailable files etc.
I suppose the better option in my case is to return null, because the result of the task is an object.
Also, I could throw checked exceptions and handle them in ThreadPoolExecutor.afterExecute (as suggested by NiranjanBhat). See Handling exceptions from Java ExecutorService tasks
Which is the better practice (in both cases)?
Perhaps there is a different way to do this or a design pattern I should use?
I would suggest that for failure during task processing, you simply throw an appropriate exception. Don't add any special handling for this in the executor. What will happen is that it will be captured, and stored in the Future. When the Future's get method is called, it will throw an ExecutionException, which the caller of get can then unpack and handle. This is essentially how normal exception handling is transposed into the Callable/Future paradigm. This looks like this:
Future<Result> futureResult = serviceClass.getResult("foo");
try {
Result result = futureResult.get();
// do something with result
}
catch (ExecutionException ee) {
Throwable e = ee.getCause();
// do something with e
}
Given that the caller of get has to have this handling of ExecutionExceptions, you could then take advantage of that to deal with failure during submission. To do this, you could construct a Future that is like Apache Commons's constantFuture, but which throws a given exception rather than returns a given value. I don't think there's anything like that in the JDK, but it's simple (if tedious) to write:
public class FailedFuture<T> implements Future<T> {
private final Throwable exception;
public FailedFuture(Throwable exception) {
this.exception = exception;
}
#Override
public T get() throws ExecutionException {
throw new ExecutionException(exception);
}
#Override
public T get(long timeout, TimeUnit unit) throws ExecutionException {
return get();
}
#Override public boolean cancel(boolean mayInterruptIfRunning) { return false; }
#Override public boolean isCancelled() { return false; }
#Override public boolean isDone() { return true; }
}
This is somewhat dodgy - you're taking a failure during a synchronously-called method, and making it look like a failure during the asynchronously-called method. You're shifting the burden of handling the error from the code that actually caused it to some code that runs later. Still, it does mean you can have all the failure handling code in one place; that might be enough of an advantage to make this worthwhile.
You can use afterExecute method. This is defined in the ThreadPoolExecutor, which you will need to override.
This method is called after the execution of each task is completed. You will get the task instance in this callback method. You can record the errors in some variable in your task and access it in this method.