We Keep Coding

Publish single item dynamically with Subscriber that has timeout

Currently we have two separate API endpoints.
public Mono<ServerResponse> get(ServerRequest request) {
Sinks.StandaloneMonoSink<String> sink = Sinks.promise();
sinkMap.putIfAbsent(randomID, sink);
return sink.asMono().timeout(Duration.ofSeconds(60))
.flatMap(val -> ServerResponse.ok().body(BodyInserters.fromValue(val)))
}
public Mono<ServerResponse> push(ServerRequest request) {
Sinks.StandaloneMonoSink<String> sink = sinkMap.remove(randomID);
if (sink == null) {
return ServerResponse.notFound().build(); }
else {
return request.bodyToMono(String.class)
.flatMap(data -> {
sink.success(data);
return ServerResponse().ok().build();
}
}
}
The intention is for client to do a get request and to keep the connection open for 1 min or so waiting for some data to arrive. And then on push request data will be published to the open connection for get and the connection will close upon receipt of first element.
The issue with current approach is that the data may be emitted after get request times out and subscription is canceled, thus losing the data. Is it possible if no subscribers then if I try to emit item throw error or perform another action with data (from the push request side).
Thanks.

I had to read this question multiple times to understand what you are looking for!
I tried something like this and it seems to work.
private final DirectProcessor<String> processor = DirectProcessor.create();
private final FluxSink<String> sink = processor.sink();
// processor has a method to check if there are any active subscribers.
// if it is true, lets push data, otherwise we can throw exception / store it somehwere
#GetMapping("/push/{val}")
public boolean push(#PathVariable String val){
boolean status = processor.hasDownstreams();
if(status)
sink.next(val);
return status;
}
#GetMapping("/get")
public Mono<String> get(){
return processor
.next()
.timeout(Duration.ofMinutes(1));
}
Question:
Will you be running only one instance of this application? What will happen when you run multiple instances of this application?
For ex: User A might push the data to app-instance-1 and User B might subscribe to app-instance-2. In this case, User B might not get data. In this case, you might need something like Redis to store this data and share among all the instances for pub/sub behavior.

Undertow handlers make all stack Non-Blocking

I'm studying undertow because I've seen is a good choice if you want to implement Non-Blocking IO and you want to have a reactive http listener.
Undertow uses handlers to handle http requests in a Non-Blocking way.
If I have some logic to be implemented between request and response, how to make this logic to be Non-Blocking too, inside of an undertow handler?
I mean, if it's inserted (or called) within the handleRequest() method is already dispatched to a working thread and then already Non-Blocking or do you need to use CompletableFuture, or Rx Observable or any other reactive library, in order to guarantee that all the stack is reactive?
This is my Handler class as a title of example, I simulate to read a Json which will be parsed into a Person.class object and the transformed (business logic) and then returned back as a Json response.
I've written the two alternatives, in order to understand better how to make the whole stack reactive and Non-Blocking.
Which one do I have to use?
public class DoBusinessLogicHandler implements HttpHandler {
JsonConverter json = JsonConverter.getInstance();
#Override
public void handleRequest(HttpServerExchange exchange) throws Exception {
if (exchange.isInIoThread()) {
exchange.dispatch(this);
return;
}
Pooled<ByteBuffer> pooledByteBuffer = exchange.getConnection().getBufferPool().allocate();
ByteBuffer byteBuffer = pooledByteBuffer.getResource();
byteBuffer.clear();
exchange.getRequestChannel().read(byteBuffer);
int pos = byteBuffer.position();
byteBuffer.rewind();
byte[] bytes = new byte[pos];
byteBuffer.get(bytes);
byteBuffer.clear();
pooledByteBuffer.free();
String requestBody = new String(bytes, Charset.forName("UTF-8") );
/* FIRST ALTERNATIVE:
you can call the business logic directly because the whole body of handleRequest() is managed reactively
*/
Person person = (Person) json.getObjectFromJson(requestBody, Person.class);
Person p = transform(person);
sendResponse(exchange, json.getJsonOf(p));
/* SECOND ALTERNATIVE
you must wrap business logic within a reactive construction (RxJava, CompletableFuture, ecc.) in order to
have all the stack reactive
*/
CompletableFuture
.supplyAsync(()-> (Person) json.getObjectFromJson(requestBody, Person.class))
.thenApply(p -> transform(p))
.thenAccept(p -> sendResponse(exchange, json.getJsonOf(p)));
}
/* it could be also a database fetch or whatever */
private Person transform(Person p){
if(p!=null){
p.setTitle(p.getTitle().toUpperCase());
p.setName(p.getName().toUpperCase());
p.setSurname(p.getSurname().toUpperCase());
}
return p;
}
private void sendResponse(HttpServerExchange exchange, String response){
exchange.getResponseHeaders()
.put(Headers.CONTENT_TYPE, "application/json");
exchange.getResponseSender()
.send(response);
}
}

How to access metadata stored by Spark Streaming custom receiver?

Spark Streaming provides the ability to create a custom receiver, as detailed here. To store the data received by the receiver into Spark, the store(data) method needs to be used.
The data I am storing to Spark has certain properties that are associated with it. The Spark Receiver class, extended by the custom receiver, provides several store methods of the form store(data, metadata), that imply that metadata/properties can be stored with the data. The code extract below shows how I used this method to store the data and its metadata/properties.
public class CustomReceiver extends Receiver<String> {
public CustomReceiver() {
super(StorageLevel.MEMORY_AND_DISK_2());
}
#Override
public void onStart() {
new Thread() {
#Override
public void run() {
try {
receive();
} catch (IOException e) {
restart("Error connecting: ", e);
}
}
}.start();
}
#Override
public void onStop() {
// Not needed as receive() method closes resources when stopped
}
private void receive() throws IOException {
String str = getData();
Map<String, String> metadata = getMetadata();
Iterator<String> it = Arrays.asList(str.split("\n\r")).iterator();
store(it, metadata);
if (isStopped()) {
closeConnections();
}
}
}
This stored data is accessed, from another class, as shown in the following code extract:
private void testCustomReceiver() {
JavaDStream<String> custom = ssc.receiverStream(new CustomReceiver());
JavaDStream<String> processedInput = custom.flatMap(row -> {
return Arrays.asList(row.split("\\r?\\n"));
});
processedInput.print();
}
Which now brings us to my question: How can the metadata/properties stored with the data in the custom receiver be accessed from the testCustomReceiver() method shown above?
I have tried searching through the documentation and exploring the JavaDStream object in the debugger to search for the metadata, but to no avail. Any help or advice on this matter would be greatly appreciated, thank you.

I've been digging around in the Spark code, and I've come to the belief that you can't ever access it again. In fact, I do not believe it is ever used.
The supervisor for your Receiver takes the metadataOption and drops it into a ReceivedBlockInfo (which is private to org.apache.spark.streaming). From there, it goes... nowhere. I can find no reference to ReceivedBlockInfo.metadataOption in the streaming codebase. It's hypothetically possible that ReceivedBlockInfo is serialized then deserialized into a different class, or some funky reflection retrieves the metadata, but both of those are such antipatterns that I wouldn't count on it happening.
Why is it there? I believe the intention was for it to be part of the Metadata Checkpointing system, but it either was never hooked up, or the connection between Receiver metadata and stream checkpointing was severed.
Either way, block metadata is gone once the block is dropped into the stream.

How do I manage request codes android with more readability?

I wish to find a better way of handling responses than the method I was taught by the company that I worked for.
I was taught to use a generic HttpClient, that used volley to send the requests. The client had a static method that would be given a generic listener, ResponseListener, which would make the callback to the context when a volley response came through. The ResponseListener would keep track of all the request "type"s . That is the code given to the client so that the context can differentiate between requests.
Is there any other way of keeping track of all the request codes without having to keep one big directory type interface file? It becomes quite overwhelming to look at when you get past 100 requests. Then you write wrapper functions for the codes in the client, and it too gets messy.
HttpClient
public static void doRequestString(final ResponseListener listener, final int type, final String url, final JSONObject postData) {
// Request a string response
StringRequest request = new StringRequest(Request.Method.GET, url,
new Response.Listener<String>() {
#Override
public void onResponse(String response) {
// Result handling
listener.onRequestDone(type, response);
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
// Error handling
System.out.println("Something went wrong!");
error.printStackTrace();
}
});
request.setTag(context);
VolleyClient.getInstance(context).getRequestQueue().add(request);
}
Listener
public interface ResponseListener
{
int HELLO_REQUEST = 0;
int GOODBYE_REQUEST = 1;
// every other request numbered here, so they don't conflict
void onRequestDone(int type, String response);
}
Context
public void onRequestDone(int type, String response)
{
switch(type) {
case Response.Listener.HELLO_REQUEST:
handleHello();
break;
case Response.Listener.GOODBYE_REQUEST:
handleGoodbye();
break;
}
}

Well, there are not that many options, to be honest. You are dealing with your responses in a centralised manner right now. Another option would be stripping the request type as having a listener per request. The main disadvantage here is that you will get your code full of listeners.
I would suggest you try and combine the two approaches in a way that suites your use case. Maybe create an intermediate service layer, divide your functionality by some of their property (for example logically - all user requests grouped together, etc), expose a single listener per service and manage the request codes there. This way you can have a bit of modularity.

how do I suspend a request in Play Framework?

I'm playing around with the Play Framework (v2.2.2), and I'm trying to figure out how to suspend an HTTP request. I'm trying to create a handshake between users, meaning, I want user A to be able to fire off a request and wait until user B "connects". Once the user B has connected, user A's request should return with some information (the info is irrelevant; let's just say some JSON for now).
In another app I've worked on, I use continuations to essentially suspend and replay an HTTP request, so I have something like this...
#Override
public JsonResponse doGet(HttpServletRequest request, HttpServletResponse response) {
Continuation reqContinuation = ContinuationSupport.getContinuation(request);
if (reqContinuation.isInitial()) {
...
reqContinuation.addContinuationListener(new ContinuationListener() {
public void onTimeout(Continuation c) {...}
public void onComplete(Continuation c) {...}
});
...
reqContinuation.suspend();
return null;
}
else {
// check results and return JsonResponse with data
}
}
... and at some point, user B will connect and the continuation will be resumed/completed in a different servlet. Now, I'm trying to figure out how to do this in Play. I've set up my route...
GET /test controllers.TestApp.test()
... and I have my Action...
public static Promise<Result> test() {
Promise<JsonResponse> promise = Promise.promise(new Function0<JsonResponse>() {
public JsonResponse apply() {
// what do I do now...?
// I need to wait for user B to connect
}
});
return promise.map(new Function<JsonResponse, Result>() {
public Result apply(JsonResponse json) {
return ok(json);
}
});
}
I'm having a hard time understanding how to construct my Promise. Essentially, I need to tell user A "hey, you're waiting on user B, so here's a promise that user B will eventually connect to you, or else I'll let you know when you don't have to wait anymore".
How do I suspend the request such that I can return a promise of user B connecting? How do I wait for user B to connect?

You need to create a Promise that can be redeemed later. Strangely, the Play/Java library (F.java) doesn't seem to expose this API, so you have to reach into the Scala Promise class.
Create a small Scala helper class for yourself, PromiseUtility.scala:
import scala.concurrent.Promise
object PromiseUtility {
def newPromise[T]() = Promise[T]()
}
You can then do something like this in a controller (note, I don't fully understand your use case, so this is just a rough outline of how to use these Promises):
if (needToWaitForUserB()) {
// Create an unredeemed Scala Promise
scala.concurrent.Promise<Json> unredeemed = PromiseUtility.newPromise();
// Store it somewhere so you can access it later, e.g. a ConcurrentMap keyed by userId
storeUnredeemed(userId, unredeemed);
// Wrap as an F.Promise and, when redeemed later on, convert to a Result
return F.Promise.wrap(unredeemed.future()).map(new Function<Json, Result>() {
#Override
public Result apply(Json json) {
return ok(json);
}
});
}
// [..]
// In some other part of the code where user B connects
scala.concurrent.Promise<Json> unredeemed = getUnredeemed(userId);
unredeemed.success(jsonDataForUserB);