We Keep Coding

Passing results from expensive methods as they come for multiple layers

I've got a code that looks similar to this:
List<String> ids = expensiveMethod();
List<String> filteredIds = cheapFilterMethod(ids);
if (!filteredIds.isEmpty()) {
List<SomeEntity> fullEntities = expensiveDatabaseCall(filteredIds);
List<SomeEntity> filteredFullEntities = anotherCheapFilterFunction(fullEntities);
if (!filteredFullEntities.isEmpty()) {
List<AnotherEntity> finalResults = stupidlyExpensiveDatabaseCall(filteredFullEntities);
relativelyCheapMethod(finalResults);
}
}
It's basically a waterfall of a couple expensive methods that, on their own, all either grab something from a database or filter previous database results. This is due to stupidlyExpensiveDatabaseCall, which needs as few leftover entities as possible, hence the exhaustive filtering.
My problem is that the other functions aren't all quite cheap either and thus they block the thread for a couple of seconds while stupidlyExpensiveDatabaseCall is waiting and doing nothing until it gets the whole batch at once.
I'd like to process the results from each method as they come in. I know I could write a thread for each individual method and have some concurrent queue working between them, but that's a load of boilerplate that I'd like to avoid. Is there a more elegant solution?

There's a post about different ways to parallelize, not only the parallelStream() way, but also that consecutive steps run in parallel the way you described, linked by queues. RxJava may suit your need in this respect. Its a more complete variety of the rather fragmentary reactive streams API in java9. But I think, you're only really there if you use a reactive db api along with it.
That's the RxJava way:
public class FlowStream {
#Test
public void flowStream() {
int items = 10;
print("\nflow");
Flowable.range(0, items)
.map(this::expensiveCall)
.map(this::expensiveCall)
.forEach(i -> print("flowed %d", i));
print("\nparallel flow");
Flowable.range(0, items)
.flatMap(v ->
Flowable.just(v)
.subscribeOn(Schedulers.computation())
.map(this::expensiveCall)
)
.flatMap(v ->
Flowable.just(v)
.subscribeOn(Schedulers.computation())
.map(this::expensiveCall)
).forEach(i -> print("flowed parallel %d", i));
await(5000);
}
private Integer expensiveCall(Integer i) {
print("making %d more expensive", i);
await(Math.round(10f / (Math.abs(i) + 1)) * 50);
return i;
}
private void await(int i) {
try {
Thread.sleep(i);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
private void print(String pattern, Object... values) {
System.out.println(String.format(pattern, values));
}
}
The maven repo:
<!-- https://mvnrepository.com/artifact/io.reactivex.rxjava2/rxjava -->
<dependency>
<groupId>io.reactivex.rxjava2</groupId>
<artifactId>rxjava</artifactId>
<version>2.2.13</version>
</dependency>

You could use CompleteableFuture to divide up each non-CPU-bound step. The usage is similar to the javascript promise API.
public void loadEntities() {
CompletableFuture.supplyAsync(this::expensiveMethod, Executors.newCachedThreadPool())
.thenApply(this::cheapFilterMethod)
.thenApplyAsync(this::expensiveDatabaseCall)
.thenApply(this::anotherCheapFilterFunction)
.thenApplyAsync(this::stupidlyExpensiveDatabaseCall)
.thenAccept(this::relativelyCheapMethod);
}
private List<String> expensiveMethod() { ... }
private List<String> cheapFilterMethod(List<String> ids) { ... }
private List<SomeEntity> expensiveDatabaseCall(List<String> ids) { ... }
private List<SomeEntity> anotherCheapFilterFunction(List<SomeEntity> entities) { ... }
private List<AnotherEntity> stupidlyExpensiveDatabaseCall(List<SomeEntity> entities) { ... }
private void relativelyCheapMethod(List<AnotherEntity> entities) { ... }
You can also pass your own thread pool at each step if you'd like to have more control over execution.

You can use Java 8 Stream API. It's impossible to process a DB query "as they come in" because the result set will come in all at once. You'd have to change your method to handle single entities.
expensiveMethod().parallelStream()
.filter(this::cheapFilterMethod) // Returns Boolean
.map(this::expensiveDatabaseCallSingle) // Returns SomeEntity
.filter(this::anotherCheapFilterFunction) // Returns boolean for filtered entities
.map(this::stupidlyExpensiveDatabaseCallSingle) // Returns AnotherEntity
.forEach(this::relativelyCheapMethod); // void method
I would also suggest using an ExecutorService to manage your threads so you don't consume all resources just creating a bunch of threads:
ExecutorService threadPool = Executors.newFixedThreadPool(8);
threadPool.submit(this::methodForParallelStream);

logic using functional-style exception handling with java and Vavr

I'm trying to get into basics of functional programming with Java 8 and I have a simple task which is to set a property on the object and then persist it. The database proper type is ltree so it might fail if it contains not allowed characters. I want to process items one-by-one and log exceptions/successes.
I choose to use the Vavr library because Try.of() exception handling and I want to learn to just use it as it seems very helpful.
here is what I came up with but I'm not satisfied enough:
public class PathHandler {
private final DocVersionDAO dao;
public void processWithHandling() {
Try.of(this::process)
.recover(x -> Match(x).of(
Case($(instanceOf(Exception.class)), this::logException)
));
}
private Stream<Try<DocVersion>> logException(Exception e) {
//log exception now but what to return? also I would like to have DocVersion here too..
return null;
}
public Stream<Try<DocVersion>> process() {
return dao.getAllForPathProcessing() //returns Stream<DocVersion>
.map(this::justSetIt)
.map(this::save);
}
public DocVersion justSetIt(DocVersion v) {
String path = Optional.ofNullable(v.getMetadata().getAdditionals().get(Vedantas.PATH))
.orElse(null);
log.info(String.format("document of uuid %s has matadata path %s; setting it", v.getDocument2().getUUID(), path));
v.getDocument2().setPath(path);
return v;
}
#Transactional(propagation = Propagation.REQUIRES_NEW)
public Try<DocVersion> save(DocVersion v) {
return Try.of(() -> dao.save(v));
}
}
the goal is quite simple so could you teach me proper way to do it?

I'm afraid, this will become highly opinionated. Anyway, I try something.
... which happened before I realized, what Vavr actually provides. It attempts to cover everything mentioned here, like immutable data structures and monad syntax sugaring (with the For statement), and goes beyond that by coming up even with pattern matching. It takes a comprehensive set of FP concepts and rebuilds them using Java and it is no surprise Scala comes into one's mind seeing this ("Vavr is greatly inspired by Scala").
Now the foundations of functional programming can't be covered by a single SO post. And it might be problematic to get familiar with them in a language like Java which isn't geared towards it. So perhaps it is better to approach them in their natural habitat like the Scala language, which is still in some proximity to Java, or Haskell, which is not.
Coming back from this detour applying the features of Vavr may be more straight foward for the initiated. But likelely not for the Java developer sitting next to you in the office, who is less willing to go the extra mile and comes up with arguments that can't be just dismissed, like this one: "If we wanted to it that way, we would be a Scala shop". Therefore I'd say, applying Vavr asks for a pragmatic attitute.
To corroborate the Vavra-Scala argument, let's take Vavra's For construct (all Lists mentioned are io.vavr.collection.List), it looks like this:
Iterator<Tuple2<Integer, String>> tuples =
For(List.of(1, 2, 3), i ->
For(List.of(4, 5, 6))
.yield(a -> Tuple.of(i, String.valueOf(a))));
In Scala you'd encounter For and yield this way.
val tuples = for {
i <- 1 to 3
a <- 4 to 6
} yield (i, String.valueOf(a))
All the monad machinery remains under the hood, where Vavra brings more of an approximation, necessarily leaking some internals. For the purpose of learning it might be puzzling to start with Vavra's hybrid creatures.
So what remains of my post is a small time treatment of some FP basics, using the example of the OP, elaborating on immutability and Try on a trench-level, but omitting pattern matching. Here we go:
One of the defining characteristics of FP are functions free of side effects ("pure functions"), which naturally (so to speak) comes along with immutable data structures/objects, which may sound kind of weird. One obvious pay off is, that you don't have to worry, that your operations create unintended changes at some other place. But Java doesn't enforce that in any way, also its immutable collections are only so on a superficial level. From the FP signature characteristics Java only offers higher order functions with java-lambdas.
I used the functional style quite a bit on the job manipulating complicated structures where I stuck to those 2 principles. E.g. load a tree T of objects from a db, do some transformations on it, which meant producing another tree of objects T', sort of one big map operation, place the changes in front of the user to accept or reject them. If accepted, apply the changes to the related JPA entities and persist them. So after the functional transformation two mutations were applied.
I'd propose, to apply FP in this sense and tried to formulate an according version of your code, using an immutable DocVersion class. I chose to simplify the Metadata part for the sake of the example.
I also tried to highlight, how the "exception-free" Try approach (some of it poached from here) could be formulated and utilized some more. Its a small time version of Vavr's Try, hopefully focusing on the essentials. Note its proximity to Java's Optional and the map and flatMap methods in there, which render it an incarnation of the FP concept called monad. It became notorious in a sweep of highly confusing blog posts some years ago usually starting with "What is a monad?" (e.g. this one). They have cost me some weeks of my life, while it is rather easy to get a good intuition of the issue just by using Java streams or Optionals. Miran Lipovaca's "Learn Yourself a Haskell For Great Good" later made good for it to some extent, and Martin Odersky's Scala language.
Boasting with of, map and flatMap, Try would, roughly speaking, qualify for a syntax-sugaring like you find it in C# (linq-expressions) or Scala for-expressions. In Java there is no equivalent, but some attempts to at least compensate a bit are listed here, and Vavr looks like another one. Personally I use the jool library occasionally.
Passing around streams as function results seems not quite canonical to me, since streams are not supposed to get reused. That's also the reason to create a List as an intermediary result in process().
public class PathHandler {
class DocVersionDAO {
public void save(DocVersion v) {
}
public DocVersion validate(DocVersion v) {
return v;
}
public Stream<DocVersion> getAllForPathProcessing() {
return null;
}
}
class Metadata {
#Id
private final Long id;
private final String value;
Metadata() {
this.id = null;
this.value = null;
}
Metadata(Long id, String value) {
this.id = id;
this.value = value;
}
public Optional<String> getValue() {
return Optional.of(value);
}
public Metadata withValue(String value) {
return new Metadata(id, value);
}
}
public #interface Id {
}
class DocVersion {
#Id
private Long id;
private final Metadata metadatata;
public Metadata getMetadatata() {
return metadatata;
}
public DocVersion(Long id) {
this.id = id;
this.metadatata = new Metadata();
}
public DocVersion(Long id, Metadata metadatata) {
this.id = id;
this.metadatata = metadatata;
}
public DocVersion withMetadatata(Metadata metadatata) {
return new DocVersion(id, metadatata);
}
public DocVersion withMetadatata(String metadatata) {
return new DocVersion(id, this.metadatata.withValue(metadatata));
}
}
private DocVersionDAO dao;
public List<DocVersion> process() {
List<Tuple2<DocVersion, Try<DocVersion>>> maybePersisted = dao.getAllForPathProcessing()
.map(d -> augmentMetadata(d, LocalDateTime.now().toString()))
.map(d -> Tuple.of(d, Try.of(() -> dao.validate(d))
.flatMap(this::trySave)))
.peek(i -> i._2.onException(this::logExceptionWithBadPracticeOfUsingPeek))
.collect(Collectors.toList());
maybePersisted.stream()
.filter(i -> i._2.getException().isPresent())
.map(e -> String.format("Item %s caused exception %s", e._1.toString(), fmtException(e._2.getException().get())))
.forEach(this::log);
return maybePersisted.stream()
.filter(i -> !i._2.getException().isPresent())
.map(i -> i._2.get())
.collect(Collectors.toList());
}
private void logExceptionWithBadPracticeOfUsingPeek(Exception exception) {
logException(exception);
}
private String fmtException(Exception e) {
return null;
}
private void logException(Exception e) {
log(fmtException(e));
}
public DocVersion augmentMetadata(DocVersion v, String augment) {
v.getMetadatata().getValue()
.ifPresent(m -> log(String.format("Doc %d has matadata %s, augmenting it with %s", v.id, m, augment)));
return v.withMetadatata(v.metadatata.withValue(v.getMetadatata().value + augment));
}
public Try<DocVersion> trySave(DocVersion v) {
return new Try<>(() -> {
dao.save(v);
return v;
});
}
private void log(String what) {
}
}
Try looks like this
public class Try<T> {
private T result;
private Exception exception;
private Try(T result, Exception exception) {
this.result = result;
this.exception = exception;
}
public static <T> Try<T> of(Supplier<T> f)
{
return new Try<>(f);
}
T get() {
if (result == null) {
throw new IllegalStateException();
}
return result;
}
public void onException(Consumer<Exception> handler)
{
if (exception != null)
{
handler.accept(exception);
}
}
public <U> Try<U> map(Function<T, U> mapper) {
return exception != null ? new Try<>(null, exception) : new Try<>(() -> mapper.apply(result));
}
public <U> Try<U> flatMap(Function<T, Try<U>> mapper) {
return exception != null ? null : mapper.apply(result);
}
public void onError(Consumer<Exception> exceptionHandler) {
if (exception != null) {
exceptionHandler.accept(exception);
}
}
public Optional<Exception> getException() {
return Optional.of(exception);
}
public Try(Supplier<T> r) {
try {
result = r.get();
} catch (Exception e) {
exception = e;
}
}
}

RxJava: How to run two sequential calls : second depends on first

I need to create an observable where 2 blocks of retrofit based calls are run sequentially. I know I can just run the second retrofit call inside one Observer call but it will be pretty messy. I have my Observable code in a separate class from the caller and it returns an Observable. I'd like to pass the result of the first call to the second then when the second call is done pass back an Observable to the calling class. (I'm using Java 7 and not 8)
public class GetFollowing {
public Observable< ArrayList<Media> > init() {
return Observable.create(
new Observable.OnSubscribe< ArrayList<Media> >() {
#Override
public void call(Subscriber<? super ArrayList<Media> > subscriber) {
...
I also need to pass back to the calling class a different type than I pass to teh second retrofit call. I been reading about map flatMap and concat but I can't seem to figure out how to structure them for my use here.
UPDATE
I came up with this, not sure if its the most elegant or if it will work at all...but if it does work is there any way to pass the result of first observable to second? Also how would I handle an issue if first observable fails?
Observable< ArrayList<Media> > test;
Observable.concat(
Observable.create(
new Observable.OnSubscribe< ArrayList<User> >() {
#Override
public void call(Subscriber<? super ArrayList<User> > subscriber) {
}
}
),
test = Observable.create(
new Observable.OnSubscribe< ArrayList<Media> >() {
#Override
public void call(Subscriber<? super ArrayList<Media> > subscriber) {
}
}
)
);
return test;

If the the requirement can be rephrased as below:
You have two methods to be executed and both return Observables.
The items emitted on first method's Observable needs to be fed into the second method as and when they occur.
The output of second method is an Observable which is based on some computation on items of first Observable.
The readily available flatMap feature in RxJava is the solution for you. Below is a simple implementation to assist you.
public static void main(String[] args) {
Observable<Integer> o1 = Observable.just(1, 2);
Observable<String> result = o1.flatMap(result1 -> Observable.just("Value is: "+result1));
result.subscribe(finalResult -> System.out.println("Final result: "+finalResult));
}
Output:
Final result: Value is: 1
Final result: Value is: 2
On the other side, if second method does not return an Observable, but performs some operation on the emitted item, you can implement the same using map.

How to create my own hot Observable from scratch?

How to create my own hot Observable from scratch?
I would like create my own function, returning observable, returning locations:
public static Observable<Location> locationObservable(Context context, String provider, long minTime, float minDistance) {
This is for Android. It is recommended to use Observable.create() for this purposes, but example shows just passing constant list of integers to each subscriber, which is not hot.
If I do something else here, for example, remember a list of subscribers, then how will I implement unsubscribing and many other features?
I.e. absolutely no idea is what to do inside Observable.OnSubscribe<Integer>() implementation?

Generally to create hot observable you use some kind of Subject: PublishSubject, BehaviorSubject, etc.
See examples for BehaviorSubject here.
class LocationService {
private Subject<Location> subject = BehaviorSubject.create();
Observable<Location> locationObservable(...) {
return subject;
}
void onNewLocationListener(Location newLocation) {
subject.onNext(newLocation);
}
}

It is not recommended to write your own, at least until you are proficient with the existing ones and need a peculiar caching/emission pattern not covered by the default 5 (Async, Behavior, Publish, Replay, Unicast).
I have a 3 part series on the subject (pun intended) if you really want to:
Part 1
Part 2
Part 3

Look at this wonderful example taken directly from Realm's RealmObservableFactory:
#Override
public Observable<Realm> from(Realm realm) {
final RealmConfiguration realmConfig = realm.getConfiguration();
return Observable.create(new Observable.OnSubscribe<Realm>() { // create new observable
#Override
public void call(final Subscriber<? super Realm> subscriber) { // this is executed on `subscribeOn(Scheduler)`
final Realm observableRealm = Realm.getInstance(realmConfig);
final RealmChangeListener<Realm> listener = new RealmChangeListener<Realm>() {
#Override
public void onChange(Realm realm) {
if (!subscriber.isUnsubscribed()) { // always check if subscriber is unsubscribed!
subscriber.onNext(observableRealm);
}
}
};
subscriber.add(Subscriptions.create(new Action0() { // add unsubscription first! thread specified by unsubscribeOn(Scheduler)
#Override
public void call() {
observableRealm.removeChangeListener(listener); // remove listener
observableRealm.close();
}
}));
observableRealm.addChangeListener(listener); // add listener
subscriber.onNext(observableRealm); // initial value
}
});
}
And read the comments, it's a pretty good example.

How can I use fetchMap() with a RecordMapper?

I know I can fetch a map something like this:
this.ctx.select(
shopSubscription.field(SHOP_SUBSCRIPTION.SHOP_ID),
shopSubscription.field(SHOP_SUBSCRIPTION.PAYMENT_GATEWAY_SUBSCRIPTION_ID),
shopSubscription.field(SHOP_SUBSCRIPTION.ADMIN_TOOL_FEATURE_TYPE_ID),
PAYMENT_GATEWAY_SUBSCRIPTION.SUBSCRIPTION_ID_TOKEN
)
.from(PAYMENT_GATEWAY_SUBSCRIPTION)
.join(shopSubscription)
.on(PAYMENT_GATEWAY_SUBSCRIPTION.ID.eq(shopSubscription.field(SHOP_SUBSCRIPTION.PAYMENT_GATEWAY_SUBSCRIPTION_ID))
.and(PAYMENT_GATEWAY_SUBSCRIPTION.PAYMENT_GATEWAY_TYPE_ID.eq(paymentGatewayType)))
.fetchMap(PAYMENT_GATEWAY_SUBSCRIPTION.PAYMENT_GATEWAY_TYPE_ID, ShopSubscriptionDTO.class);
but to detect issues at compile time I'd prefer if I could additionally add a RecordMapper to this query.
So is there a way to call fetchMap() but also provide a RecordMapper?
What I'm thinking of would look something like this:
this.ctx.select(
shopSubscription.field(SHOP_SUBSCRIPTION.SHOP_ID),
shopSubscription.field(SHOP_SUBSCRIPTION.PAYMENT_GATEWAY_SUBSCRIPTION_ID),
shopSubscription.field(SHOP_SUBSCRIPTION.ADMIN_TOOL_FEATURE_TYPE_ID),
PAYMENT_GATEWAY_SUBSCRIPTION.SUBSCRIPTION_ID_TOKEN
)
.from(PAYMENT_GATEWAY_SUBSCRIPTION)
.join(shopSubscription)
.on(PAYMENT_GATEWAY_SUBSCRIPTION.ID.eq(shopSubscription.field(SHOP_SUBSCRIPTION.PAYMENT_GATEWAY_SUBSCRIPTION_ID))
.and(PAYMENT_GATEWAY_SUBSCRIPTION.PAYMENT_GATEWAY_TYPE_ID.eq(paymentGatewayType)))
// For each record apply the map() function
.map(new RecordMapper<Record<?>, ShopSubscriptionDTO>() {
#Override
public ShopSubscriptionDTO map(Record<?> record) {
ShopSubscriptionDTO shopSubscriptionDto = new ShopSubscriptionDTO();
shopSubscriptionDto.setShopId(record.getValue(SHOP_SUBSCRIPTION.SHOP_ID)
// ...
return shopSubscriptionDto;
}
});
// Fetch the result into a map where the key is SHOP_SUBSCRIPTION.ADMIN_TOOL_FEATURE_TYPE_ID
.fetchMap(SHOP_SUBSCRIPTION.ADMIN_TOOL_FEATURE_TYPE_ID);

Since there are quite a lot of different implementations of fetchMap() I didn't see that there is fetchMap(Field<K>, RecordMapper<? super R, R>) too. So just going with that helps solving this issue:
// ...
.fetchMap(ADMIN_TOOL_ADD_ON.ADMIN_TOOL_ADD_ON_TYPE_ID, new RecordMapper<Record, AdminToolAddOnDTO>() {
#Override
public AdminToolAddOnDTO map(Record record) {
AdminToolAddOnDTO dto = new AdminToolAddOnDTO();
dto.setId(record.getValue(ADMIN_TOOL_ADD_ON.ID));
dto.setAdminToolFeatureTypeId(record.getValue(ADMIN_TOOL_ADD_ON.ADMIN_TOOL_FEATURE_TYPE_ID));
dto.setAdminToolAddOnTypeId(record.getValue(ADMIN_TOOL_ADD_ON.ADMIN_TOOL_ADD_ON_TYPE_ID));
dto.setPrice(record.getValue(ADMIN_TOOL_ADD_ON.PRICE));
dto.setCountryId(record.getValue(ADMIN_TOOL_ADD_ON.COUNTRY_ID));
dto.setAddOnIdToken(record.getValue(ADMIN_TOOL_ADD_ON_TYPE.ADD_ON_ID_TOKEN));
return dto;
}
});

java 8 or higher
.fetchMap(CN_TASKS.AGENTID,
r -> new CnTaskMessage(r.getValue(CN_TASKS.CN_TASKID), r.getValue(CN_TASKS.TASK_TYPE),
r.getValue(CN_TASKS.STATUS)));