We Keep Coding

How do I create a Flux where each element of the flux is a polling operation and terminate the flux on success or error?

Similar to How to process each product one by one with incremental progress update using Spring reactive?
The thing I want to do is given
enum Status {
PROCESSING,
ERROR,
COMPLETE
}
record MyResp {
String requestId;
Status status;
double progress;
URI result;
String errorMessage;
};
Mono<MyResp> requestSomethingSlow(MyReqObjectNotRelevant request);
/**
* Sets status to SUCCESS or return a Mono.error()
*/
Mono<MyResp> checkIfDone(String requestId);
I want a method like:
Flux<MyResp> requestSomethingSlowFlux(MyReqObjectNotRelevant request, Duration delay) {
return ...???
??? requestSomethingSlow(request)
. ???
.delayElements(delay)
. ???
. checkIfDone(...?)
...???
}
I am thinking it's something like Flux.generate but how do I convert the Mono<MyResp> to a Callable and use it with the generator function?
I was also looking at Mono.expand as shown in How do I create a Flux using generate wrapping calls that return a Mono but that wouldn't work because I don't have a finite set of calls.
So far my implementation attempt looks like
Flux<MyResp> requestSomethingSlowFlux(MyReqObjectNotRelevant request, Duration delay) {
return requestSomethingSlow(request)
.flatMapMany(initialResponse -> {
if (initialResponse.getStatus() == COMPLETE) {
return Flux.just(initialResponse);
} else if (initialResponse.getStatus() == ERROR) {
return Flux.error(
new IllegalStateException(
initialResponse.getErrorMessage()));
} else {
... still figuring this part out
}
}
.delayElements(delay)
}
Also similar to How do you implement Polling Logic in Project Reactor? but I want it as a flux of events that show progress rather than all or none.

Using this answer but removing the last() call so I get all events.
Flux<MyResp> requestSomethingSlowFlux(MyReqObjectNotRelevant request, Duration delay) {
return requestSomethingSlow(request)
.flatMapMany(initialResponse -> {
if (initialResponse.getStatus() == COMPLETE) {
return Flux.just(initialResponse);
} else if (initialResponse.getStatus() == ERROR) {
return Flux.error(
new IllegalStateException(
initialResponse.getErrorMessage()));
} else {
return checkIfDone(initialResponse.getRequestId())
.repeatWhen(repeat -> repeat.delayElements(delay))
.doNext(response -> {
if (response.getStatus() == ERROR) {
throw new IllegalStateException(
initialResponse.getErrorMessage());
}
})
.takeUntil(response -> response.getStatus() != PROCESSING)
}
});
}
It has a few flaws though
the status processing is more or less repeated from the initial response
the delay is part of the Mono and not part of the Flux. Therefore, I cannot make it return the flux and have the delayElements on the flux. I think it's the choice of repeatWhen

Don't re-execute Retrofit call if it's still in progress using RxJava 2

My goal is a method that will execute a network call unless the network call is already in progress in which case the caller subscribes to the results of the existing call.
Here is what I have, it mostly works:
private AsyncSubject<Stuff> asyncSubject;
public Observable<Stuff> getStuff() {
if (asyncSubject == null) {
asyncSubject = AsyncSubject.create();
asyncSubject
.doAfterTerminate(new Action() {
#Override
public void run() throws Exception {
asyncSubject = null;
}
})
.subscribe();
retrofitApi.getStuff()
.subscribe(asyncSubject);
}
return asyncSubject
.someOperatorsHere();
}
This mostly works because asyncSubject nulls itself out on terminate so that any subsequent calls will re-execute the network request.
But there is a bug, there's a window between if (asyncSubject == null) and return asyncSubject where doAfterTerminate can execute and I get a null pointer.
Any ideas how to fix the bug. Maybe there is a more elegant Rx way to do this? Or synchronized block. Or a way to query Retrofit for network progress.

This solution shares the subscription between ongoing subscribers.
private Observable<Stuff> stuff = retrofitApi.getStuff().share();
public Observable<Stuff> getStuff() {
return stuff
.someOperatorsHere();
}

There are couple of possible solutions, I'll just describe the one that is probably simplest in your case.
You have 2 problems: race condition, and nullifying and recreating subject, which is rather unsafe.
What you can do is save your Subscription here
retrofitApi.getStuff().subscribe(asyncSubject);
And instead of checking if (asyncSubject == null), you can check if (s.isUnsubscribed()).
So your code will look more or less like this:
private AsyncSubject<Stuff> asyncSubject = AsyncSubject.create();
private Subscription subscription = Subscriptions.unsubscribed();
public Observable<Stuff> getStuff() {
if (subscription.isUnsubscribed()) {
subscription = retrofitApi.getStuff()
.subscribe(asyncSubject);
}
return asyncSubject
.someOperatorsHere();
}

How to escalate top-most supervisors in Akka?

I have the following top-level (“parent-most”) actor:
// Groovy pseudo-code
class Master extends UntypedActor {
ActorRef child1
ActorRef child2
ActorRef child3
ActorRef backup
#Override
void onReceive(Object message) throws Exception {
if(message instanceof Terminated) {
Terminated terminated = message as Terminated
if(terminated.actor != backup) {
terminated.actor = backup
} else {
// TODO: What to do here? How to escalate from here?
}
} else {
child1.tell(new DoSomething(message), getSelf())
child2.tell(new DoSomethingElse(message), getSelf())
child3.tell(new DoSomethingElser(message, getSelf())
}
}
#Override
SupervisorStrategy supervisorStrategy() {
new OneForOneStrategy(10, Duration.minutes(“1 minute”, new Future<Throwable, SupervisorStrategy.Directive> {
#Override
Directive apply(Throwable t) throws Exception {
if(isRecoverable(t) { // Don’t worry about how/where this is defined or how it works
SupervisorStrategy.stop()
} else {
SupervisorStrategy.escalate()
}
}
})
}
}
As you can see, it supervises three children, and when those 3 children throw “recoverable” exceptions, they are stopped and are replaced with a backup. So far, so good.
The problem I’m now facing is that if the backup actors throws any throwable whatsoever, I want to consider this Master actor (and really, my app in general) to be in a state where it cannot continue processing any input, and to escalate the exception to the guardian-level.
I’m brand new to Akka and not sure where to put this code, and what it should look like. Again, I just need logic that says:
If the backup actor throws any throwable, escalate the exception to the Master’s parent, which should really be an Akka “guaradian” actor/construct
The first part of this is that we need to know when an exception is thrown from the backup; I can handle this part, so let’s pretend our strategy now looks like this:
#Override
SupervisorStrategy supervisorStrategy() {
new OneForOneStrategy(10, Duration.minutes(“1 minute”, new Future<Throwable, SupervisorStrategy.Directive> {
#Override
Directive apply(Throwable t) throws Exception {
if(wasThrownFromBackup(t)) {
SupervisorStrategy.escalate()
} else if(isRecoverable(t) {
SupervisorStrategy.stop()
} else {
SupervisorStrategy.escalate()
}
}
})
}
But as you can see, I’m still struggling to implement the escalation “out of the actor system”. Ideas? Java code example greatly preferred as Scala looks like hieroglyphics to me.

Have a look at the 'Reaper' pattern here http://letitcrash.com/post/30165507578/shutdown-patterns-in-akka-2 Sorry it is in Scala but I think it is easy enough to translate to Java.
Also have a look here, https://groups.google.com/forum/#!topic/akka-user/QG_DL7FszMU
You should set in your configuration
akka.actor.guardian-supervisor-strategy = "akka.actor.StoppingSupervisorStrategy"
This will cause any 'top level' actor which escalates to be stopped by the system. Then you implement another top level actor called 'Reaper' (or whatever you want to call it) which has just one job, watch the main top level actor and take action (e.g. context.system.shutdown()) when the top level actor stops.
I don't know the akka java API so can't provide you with an exact example, but in Scala, from the LetItCrash blog above, it looks like:
import akka.actor.{Actor, ActorRef, Terminated}
import scala.collection.mutable.ArrayBuffer
object Reaper {
// Used by others to register an Actor for watching
case class WatchMe(ref: ActorRef)
}
abstract class Reaper extends Actor {
import Reaper._
// Keep track of what we're watching
val watched = ArrayBuffer.empty[ActorRef]
// Derivations need to implement this method. It's the
// hook that's called when everything's dead
def allSoulsReaped(): Unit
// Watch and check for termination
final def receive = {
case WatchMe(ref) =>
context.watch(ref)
watched += ref
case Terminated(ref) =>
watched -= ref
if (watched.isEmpty) allSoulsReaped()
}
}
class ProductionReaper extends Reaper {
// Shutdown
def allSoulsReaped(): Unit = context.system.shutdown()
}
In your application startup, you create your master actor, create your reaper, send a WatchMe(masterActor) message to the reaper.

Handling blocking operations in Play 2.1

I am trying to create a way to handle blocking operations in a specific way in Play. First I have described what my aim is followed by what I have managed so far. Can you please tell me if I am on the right track - if so, could you help me understand how to complete the code? If it is not the right way to do it could you suggest a better alternative?
Thanks a lot for all your help
Aim:
Would like to have all blocking operations sent to one thread to a separate thread to be handled asynchronously. New requests that come in are not to take up more threads but instead place them in a queue (or anything similar) to be handled by the single thread. For each item that is processed asynchronously by the extra thread, some text must be gathered and returned to the browser.
So after reading docs and SO questions it appears that actors must be used. I like the concept of actors but have never used them before so am still learning. This is what I have:
package models;
import java.io.*;
import play.mvc.*;
import play.libs.*;
import play.libs.F.*;
import akka.actor.*;
public class ActorTest extends UntypedActor {
static BufferedReader reader = new BufferedReader(new InputStreamReader(
System.in));
#Override
public void onReceive(Object message) throws Exception {
if (message instanceof String) {
getSender().tell(
"You sent me " + ((String) message)
+ " and the consol replied with "
+ reader.readLine(), getSelf());
} else
unhandled(message);
}
}
As you can see the blocking operation is readLine() - just an way of testing.
Is this how it should be done? If so, I had assumed that from the controller, I some how create an async result or something using promises. [ Handling asynchronous results ].
Couple issues, how do I send a message to the Actor and get the reply? I mean can I get the result from a tel() call?
How do I make sure that more threads don't get taken up and that all operations go into a queue - or is this already handled by the actor?
Could you please provide an example controller action that could do this?
Your help is greatly appreciated.
PS FYI I am really new to all this so just to get to this stage I have found these docs useful to read - the Akka actor pages, play of course and some wiki pages on actors.
[edit]
sorry I said single thread but it could be a thread pool - just as long as only the assigned thread / thread pool is used to handle the blocking io not any others.

You can send a message to the Akka actor using ask (instead of tell). It will return to you a Future, which then you can map to a Promise<Result>.
However, you don't really need to use Akka if you don't have to. You can simply use Futures/Promises to run your blocking operation in the background.
In either approach, you end up with a Future from which you can complete the request when the future finishes.
Example of Using Promise in Play 2.2.x
...
import play.libs.F.*;
public static Promise<Result> index() {
Promise<Integer> promiseOfInt = Promise.promise(
new Function0<Integer>() {
public Integer apply() {
// long-running operation (will run in separate thread)
return 42;
}
});
return promiseOfInt.map(
new Function<Integer, Result>() {
public Result apply(Integer i) {
// 'i' is the result after Promise is complete
return ok("Got result: " + i);
}
});
}
If you're using Akka, you need to convert the Future returned from ask to Play's Promise as follows:
public static Promise<Result> index() {
ActorRef myActor = Akka.system().actorFor("user/my-actor");
return Promise.wrap(ask(myActor, "hello", 1000)).map(
new Function<Object, Result>() {
public Result apply(Object response) {
return ok(response.toString());
}
});
}
Example of Using Promise in Play 2.1.x
...
import play.libs.F.*;
public static Result index() {
Promise<Integer> promiseOfInt = play.libs.Akka.future(
new Callable<Integer>() {
public Integer call() {
// long-running operation (will run in separate thread)
return 42;
}
});
return async(
promiseOfInt.map(
new Function<Integer,Result>() {
public Result apply(Integer i) {
// 'i' is the result after Promise is complete
return ok("Got result: " + i);
}
}));
}
If you're using Akka, you need to convert the Future returned from ask to Play's Promise as follows:
public static Result index() {
ActorRef myActor = Akka.system().actorFor("user/my-actor");
return async(
Akka.asPromise(ask(myActor,"hello", 1000)).map(
new Function<Object,Result>() {
public Result apply(Object response) {
return ok(response.toString());
}
}
)
);
}

How can I make this thread safe?

I have a server that receives various xml messages from clients (one thread per client) and routes the messages to different functions depending on the message type. Eg. if the first element in the messages contains the string 'login' it signifies that this is a login message so route the message to the login() function.
Anyway, I want to make this message so things don't get messed up if multiple clients are connected and the dispatcher switches threads in middle of the message routing. So here is how I am routing the messages -
public void processMessagesFromClient(Client client)
{
Document message;
while (true)
{
try
{
message = client.inputStream.readObject();
/*
* Determine the message type
*/
String messageType = getMessageType(message);
// Route the message depending on its type
switch (messageType)
{
case LOGIN:
userModel.handleLogin();
...
...
...
etc...
}
} catch(Exception e) {}
}
So how can I make this thread safe? I figure I need to put a synchronise statement in somewhere but Im not sure where. Also Ive been reading around on the subject and I found this post which says there is an issue with using synchronise on 'this' -
https://stackoverflow.com/a/416198/1088617
And another post here which says singletons aren't suitable for using synchronise on (My class in the code above is a singleton) - https://stackoverflow.com/a/416202/1088617

Your class is already thread safe, because you are only using local variables.
Thread safety only comes into play when you access class state (ie fields), which your code doesn't (seem to) do.
What you are talking about is serialization - you want to funnel all message processing through one point to guarantee that message processing is one-at-a-time (starts and finishes atomically). The solution is simple: Employ a static synchronized method:
public void processMessagesFromClient(Client client) {
Document Message;
while (true) {
processMessage(client);
}
}
private static synchronized processMessage(Client client) {
try {
message = client.inputStream.readObject();
String messageType = getMessageType(message);
// Route the message depending on its type
switch (messageType) {
case LOGIN:
userModel.handleLogin();
...
etc...
}
} catch(Exception e) {}
}
FYI static synchronized methods use the Class object as the lock. This code will make your code behave like a single thread, which your question seems to want.

I would actually have a message handler thread which is responsible for reading incoming messages. This will then hand off processing to a worker thread to do the time consuming processing of the message. You can use the Java ThreadPoolExecutor to manage this.

If you already have 1 thread per connection, then the only thing that you have to synchronize are the functions which handle the events (i.e. functions like userModel.handleLogin()).

I guess the best solution should be to use a thread safe queue like the ConcurrentQueue and use a single working thread to pick up this values and run the actions one by one.

Provided you have one of these objects per thread, you don't have a problem. You only need to synchronized a shared object which can be modified by one of the threads.
public void processMessagesFromClient(Client client) {
while (true) {
processMessage(client);
}
}
private void processMessage(Client client) {
try {
Document message = client.inputStream.readObject();
String messageType = getMessageType(message);
// Route the message depending on its type
switch (messageType) {
case LOGIN:
userModel.handleLogin();
...
etc...
}
} catch(Exception e) {}
}

You need to know which resource should be only used be one thread at a certain time.
In your case it is likely that reading the next message needs to protected.
synchronize (lock) {
message = client.inputStream.readObject();
}
However, your code sample does not really show what needs to protected against concurrent access

The method itself is thread safe.
However, noting that this your class is a singleton, you might want to use double checked locking in your getInstance to ensure thread safety.
Also you should make sure your instance is set to static
class Foo {
private static volatile Foo instance = null;
public static Foo getInstance() {
if (instance == null)
{
synchronized(this)
{
if (instance == null)
instance = new Foo ();
}
}
return instance ;
}
}