I'm starting using java 8 stream API.
I would like to convert a list of "sql result set" to domain objects, i.e composite structure.
Domain objects : a user has a collection of permissions, each permission has a collection of year of applications.
For example, John has 2 permissions (MODERATOR and DEV).
its moderator permission is only applicable for 2014 and 2015
its dev permission of only applicable for 2014.
class User {
// some primitives attributes
List<Permission> permission;
}
class Permission {
// some primitives attributes
List<Integer> years;
}
Now I make a query and got a list of flat results, something like:
[1, "moderator", 2014]
[1, "moderator", 2015]
[1, "dev", 2014]
[2, "dev", 2010]
[2, "dev", 2011]
[2, "dev", 2012]
The 1 and 2 are userId.
I tried various construction but at the end it's more complex than fluent. And didn't work :)
I read in a Java 8 book that it's "simple" to build dompain objects with collectors.
I cried a little when I read that :'(
I tried
sb.collect(
collectingAndThen(
groupingBy(
Mybean::getUserId,
collectingAndThen(
groupingBy(Monbean::getPermissionId, mapping(convertPermission, toList())),
finisherFonction)
),
convertUser)
);
and got one hell of generics compilation failure.
what's the best way to construct multi level composite domain objects using java 8 streams ?
is collectionAndThen / finisher a good idea?
or do you use only groupingBy followed by a mapping function?
do you transform the classifier to an object (sort of first level mapping function?)
Because at the end I want to get rid of the Map and got a List<User> result (I think I can add a map call on the entrySet to finish the transformation).
Let me offer you a few options and you decide which looks most clear to you. I am assuming that User constructor is User(int userId, List<Permission> permissions) and Permission constructor is Permission(String permissionId, List<Integer> years)
Option 1: The direct approach. Group by userid, construct a list of permissions for each userid and make User objects. Personally, I find this much nesting in collectors to be hard to follow.
List<User> users = beans.stream()
.collect(
groupingBy(
MyBean::getUserid,
collectingAndThen(
groupingBy(
MyBean::getPermission,
mapping(MyBean::getYear, toList())
),
t -> t.entrySet().stream()
.map(e -> new Permission(e.getKey(), e.getValue()))
.collect(toList())
)
)
).entrySet().stream()
.map(e -> new User(e.getKey(), e.getValue()))
.collect(toList());
Option 2: Same as above but make the permission collector separately for clarity.
Collector<MyBean, ?, List<Permission>> collectPermissions = collectingAndThen(
groupingBy(MyBean::getPermission, mapping(MyBean::getYear, toList())),
t -> t.entrySet().stream()
.map(e -> new Permission(e.getKey(), e.getValue()))
.collect(toList())
);
List<User> users = beans.stream()
.collect(groupingBy(MyBean::getUserid, collectPermissions))
.entrySet().stream()
.map(e -> new User(e.getKey(), e.getValue()))
.collect(toList());
Option 3: First roll the beans into a map of userid to map of permissionid to list of years (Map<Integer, Map<String, List<Integer>>). Then construct the domain objects out of the map
List<User> users = beans.stream().collect(
groupingBy(
MyBean::getUserid,
groupingBy(
MyBean::getPermission,
mapping(MyBean::getYear, toList())
)
)
).entrySet().stream()
.map(u -> new User(
u.getKey(),
u.getValue().entrySet().stream()
.map(p -> new Permission(p.getKey(), p.getValue()))
.collect(toList())
)
).collect(toList());
The collectingAndThen combinator is designed for when the intermediate form for accumulation differs from the desired final form. This is the case for the joining() collector (where the intermediate form is a StringBuilder, but the final form is a String), and also can be used to wrap mutable collections with immutable wrappers after the collecting is done. So I don't think this is the tool you are looking for.
If you're looking for "permissions by user", this will do the trick:
Map<UserId, List<Permission>>
queryResults.stream()
.collect(qr -> qr.getId(),
mapping(qr -> qr.getPermission() + ":" + qr.getDate(),
toList()));
This would result in:
1 -> [ moderator:2014, moderator:2015, dev:2014 ]
2 -> [ dev:2010, dev:2011, dev:2012 ]
The idea is:
- You are grouping by "id" (the first field of your query)
- For each record, you select some function of the remaining fields (here, I used string concat for clarity, but you could create an object holding the permission and year), and send that to the downstream collector
- Use Collectors.toList() as the downstream collector, which will then be the "value" of the Map created by groupingBy.
Related
I am trying to make a service that will calculate statistics for each month.
I did smth like this:
public Map<String, BigDecimal> getStatistic() {
List<Order> orders = orderService.findAll(Sort.by(Sort.Direction.ASC, "creationDate")).toList();
SortedMap<String, BigDecimal> statisticsMap = new TreeMap<>();
MathContext mc = new MathContext(3);
for (Order order : orders) {
List<FraudDishV1Response> dishesOfOrder = order.getDishIds()
.stream()
.map(dishId -> dishV1Client.getDishById(dishId))
.collect(Collectors.toList());
BigDecimal total = calculateTotal(dishesOfOrder);
String date = order.getCreatedDate().format(DateTimeFormatter.ofPattern("yyyy-MM"));
statisticsMap.merge(date, total, (a, b) -> a.add(b, mc));
}
return statisticsMap;
}
But it takes a long time if there are lots of etries in the database.
Are there any best practices for working with statistics in REST API applications?
And also I'd like to know if it is a good way to save the statistics in a separate repository? It will save time for calculating statistics, but during creating a record in the database, you will also have to update the statistics db.
With your approach you'll eventually run out of memory while trying to load huge amount of data from database. You could do processing in batches but then again it will only get you so far. Ideally any kind of statistical data or on demand reporting would be served by long running scheduled jobs which will periodically do processing in the background and generate the desired data for you. You could dump the result in a table and then serve it from there via an API.
Another approach is to do real time processing. If you could develop a streaming pipeline in your application then I would highly suggest you may explore Apache Flink project.
Well, I did't stop and made several solutions step by step...
Step 1: Use streams. Before that, calculating statistics for 10,000 OrderEntities records took 18 seconds. Now it has accelerated to 14 seconds.
Step 2: Using parallelStream instead of streams. Parallel streams accelerated the calculation of statistics to 6 seconds! I was even surprised.
public SortedMap<String, BigDecimal> getStatisticsByParallelStreams() {
List<OrderEntity> orders = new ArrayList<>();
orderService.findAll(Sort.by(Sort.Direction.ASC, "createdDate")).forEach(orders::add);
MathContext mc = new MathContext(3);
return orders.stream().collect(Collectors.toMap(
order -> order.getCreatedDate().format(DateTimeFormatter.ofPattern("yyyy-MM")),
order -> calculateTotal(order.getDishIds()
.parallelStream()
.map(dishId -> dishV1Client.getDishById(dishId))
.collect(Collectors.toList())),
(a, b) -> a.add(b, mc),
TreeMap::new
));
}
Step 3: Optimizing requests to another microservice. I connected the JProfiler to the app and I have found out that I offen do extra requests to the another microservice. After it firstly I made a request to receive all Dishes, and then during calculating statistics, I use a recieved List of Dishes.
And thus I speeded it up to 1.5 seconds!:
public SortedMap<String, BigDecimal> getStatisticsByParallelStreams() {
List<OrderEntity> orders = new ArrayList<>();
orderService.findAll(Sort.by(Sort.Direction.ASC, "createdDate")).forEach(orders::add);
List<FraudDishV1Response> dishes = dishV1Client.getDishes();
MathContext mc = new MathContext(3);
return orders.stream().collect(Collectors.toMap(
order -> order.getCreatedDate().format(DateTimeFormatter.ofPattern("yyyy-MM")),
order -> calculateTotal(order.getDishIds()
.parallelStream()
.map(dishId -> getDishResponseById(dishes, dishId))
.collect(Collectors.toList())),
(a, b) -> a.add(b, mc),
TreeMap::new
));
}
I have this current logic:
List<String> priceUnitCodes = ofNullable(product.getProductPrices())
.map(ProductPrices::getProductPrices)
.flatMap(productPrices -> productPrices.stream()) // << error highlight
.map(ProductPrice::getPriceBase)
.map(PriceBase::getPriceUnit)
.map(UniversalType::getCode)
.collect(Collectors.toList());
Where in IntelliJ the flatMap part is highlighted and shows the following error hint:
no instance(s) of type variable(s) U exist so that Stream<ProductPrice> conforms to Optional<? extends U>
I know that Optionals and Stream are two different things but I wonder if there is a way to combine them so I can follow up an Optional<List<?>> with a Stream afterwards.
If you're on Java 9+, you can use Optional.stream, followed by flatMap:
ofNullable(product.getProductPrices())
.map(ProductPrices::getProductPrices)
.stream()
.flatMap(Collection::stream) //assuming getProductPrices returns a Collection
...
Optional.stream returns an empty stream if the optional is empty.
Since you are starting with an Optional, you have to decide what to return when that Optional is empty.
One way is to put the Stream pipeline inside the Optional's map:
List<String> priceUnitCodes = ofNullable(product.getProductPrices())
.map(ProductPrices::getProductPrices)
.map(productPrices -> productPrices.stream()
.map(ProductPrice::getPriceBase)
.map(PriceBase::getPriceUnit)
.map(UniversalType::getCode)
.collect(Collectors.toList())
.orElse(null);
Or course, if the map operations inside the Stream pipeline may return null, additional changes will be required (to avoid NullPointerException).
On the other hand, if they can never return null, they can be chained into a single map:
List<String> priceUnitCodes = ofNullable(product.getProductPrices())
.map(ProductPrices::getProductPrices)
.map(productPrices -> productPrices.stream()
.map(pp -> pp.getPriceBase().getPriceUnit().getCode())
.collect(Collectors.toList())
.orElse(null);
An alternate solution would be to get the value of the Optional using orElse and this can be done without upgrading to Java-9. It would look like:
List<String> priceUnitCodes = Optional.ofNullable(product.getProductPrices())
.map(ProductPrices::getProductPrices)
.orElse(Collections.emptyList()) // get the value from Optional
.stream()
.map(ProductPrice::getPriceBase)
.map(PriceBase::getPriceUnit)
.map(UniversalType::getCode)
.collect(Collectors.toList());
I get an PublicException: Duplicate Keys error in this place.
Map<BgwContract, List<Fee>> bgwContractFeeMap = bgwContractList
.stream()
.filter(bgwContract -> !bgwContract.getStatus().equals(BgwContractStatus.CLOSED))
.filter(bgwContract -> availableIbans.contains(bgwContract.getFeeAccount()))
.collect(
Collectors.toMap(bgwContract -> bgwContract,
bgwContractFeeService::getContractMonthlyFees)
);
I understand that the issue is that there are some duplicates and it immediately crashes. I know that a .distinct() would fix this error, but I don't want to lose any data. Is there a way how to enhance this mapping to fix this error without loosing any values, maybe some kind of a filter or any other kind of java 8 methods? I'm not talking about MultiMaps etc.
You need to pass a merge function to Collectors.toMap(), which handles values having the same key:
Map<BgwContract, List<Fee>> bgwContractFeeMap = bgwContractList
.stream()
.filter(bgwContract -> !bgwContract.getStatus().equals(BgwContractStatus.CLOSED))
.filter(bgwContract -> availableIbans.contains(bgwContract.getFeeAccount()))
.collect(
Collectors.toMap(Function.identity(),
bgwContractFeeService::getContractMonthlyFees,
(l1,l2)->{
l1.addAll(l2);
return l1;
})
);
In this case, the elements of two value lists having the same key will be concatenated into a single list.
I have a task: Given a list of users each one with some privileges, group users by privileges. I should use the stream API. This is the only code, which came to my mind:
public Map<Privilege, List<User>> groupByPrivileges(List<User> users) {
return users.stream().collect(groupingBy(User::getPrivileges));
}
As you can see in the return type of the method, I need to get a map with single Privilege objects as keys, but a User may appear multiple times, under different keys, if they have multiple privileges.
The main problem is that User::getPrivileges returns List<Privilege>, and I know only how to process a single Privilege.
You can create an entry for each privilege in the list and then group by when you collect as :
public Map<Privilege, List<User>> groupByPrivileges(List<User> users) {
return users.stream()
.flatMap(user -> user.getPrivileges().stream()
.map(privilege -> new AbstractMap.SimpleEntry<>(privilege, user)))
.collect(Collectors.groupingBy(Map.Entry::getKey,
Collectors.mapping(Map.Entry::getValue, Collectors.toList())));
}
I am new to java 8 lamdas. I need help on the below collection aggregation and sorted list.
I have Arraylist of object which has List
[
[parentName1,parentkey1, child1Name1, childKey1],
[parentName1,parentkey1, child1Name2, childKey2],
[parentName3,parentkey3, child1Name3, childKey3],
[parentName3,parentkey3, child1Name4, childKey4],
[parentName5,parentkey5, child1Name5, childKey5],
[parentName5,parentkey5, child1Name6, childKey6]
]
I would like aggregate above collections using java 8 lambdas in to sorted list.
List(Parent(parentName1,parentkey1, List( Child(child1Name1,childKey1),Child(child1Name2, childKey2) ),
Parent(parentName3,parentkey3, List( Child(child1Name3,childKey3),Child(child1Name4, childKey4) ),
Parent(parentName5,parentkey5, List( Child(child1Name5,childKey5),Child(child1Name5, childKey6) )
);
Any help would appreciated.
I did not understand sorting criteria using which you wanted to sort the list but here is generic example but with little modification it ll work for your case also.
employeeList.stream()
.sorted((e1, e2) -> Integer.compare(e1.getEmployeeNumber(), e2.getEmployeeNumber()))
.forEach(e -> System.out.println(e));
Above example shows examplyee object can be sorted based employee number, but you can change the logic to fit your case.