When would you use collect() vs reduce()? Does anyone have good, concrete examples of when it's definitely better to go one way or the other?
Javadoc mentions that collect() is a mutable reduction.
Given that it's a mutable reduction, I assume it requires synchronization (internally) which, in turn, can be detrimental to performance. Presumably reduce() is more readily parallelizable at the cost of having to create a new data structure for return after every step in the reduce.
The above statements are guesswork however and I'd love an expert to chime in here.
reduce is a "fold" operation, it applies a binary operator to each element in the stream where the first argument to the operator is the return value of the previous application and the second argument is the current stream element.
collect is an aggregation operation where a "collection" is created and each element is "added" to that collection. Collections in different parts of the stream are then added together.
The document you linked gives the reason for having two different approaches:
If we wanted to take a stream of strings and concatenate them into a
single long string, we could achieve this with ordinary reduction:
String concatenated = strings.reduce("", String::concat)
We would get the desired result, and it would even work in parallel.
However, we might not be happy about the performance! Such an
implementation would do a great deal of string copying, and the run
time would be O(n^2) in the number of characters. A more performant
approach would be to accumulate the results into a StringBuilder,
which is a mutable container for accumulating strings. We can use the
same technique to parallelize mutable reduction as we do with ordinary
reduction.
So the point is that the parallelisation is the same in both cases but in the reduce case we apply the function to the stream elements themselves. In the collect case we apply the function to a mutable container.
The reason is simply that:
collect() can only work with mutable result objects.
reduce() is designed to work with immutable result objects.
"reduce() with immutable" example
public class Employee {
private Integer salary;
public Employee(String aSalary){
this.salary = new Integer(aSalary);
}
public Integer getSalary(){
return this.salary;
}
}
#Test
public void testReduceWithImmutable(){
List<Employee> list = new LinkedList<>();
list.add(new Employee("1"));
list.add(new Employee("2"));
list.add(new Employee("3"));
Integer sum = list
.stream()
.map(Employee::getSalary)
.reduce(0, (Integer a, Integer b) -> Integer.sum(a, b));
assertEquals(Integer.valueOf(6), sum);
}
"collect() with mutable" example
E.g. if you would like to manually calculate a sum using collect() it can not work with BigDecimal but only with MutableInt from org.apache.commons.lang.mutable for example. See:
public class Employee {
private MutableInt salary;
public Employee(String aSalary){
this.salary = new MutableInt(aSalary);
}
public MutableInt getSalary(){
return this.salary;
}
}
#Test
public void testCollectWithMutable(){
List<Employee> list = new LinkedList<>();
list.add(new Employee("1"));
list.add(new Employee("2"));
MutableInt sum = list.stream().collect(
MutableInt::new,
(MutableInt container, Employee employee) ->
container.add(employee.getSalary().intValue())
,
MutableInt::add);
assertEquals(new MutableInt(3), sum);
}
This works because the accumulator container.add(employee.getSalary().intValue()); is not supposed to return a new object with the result but to change the state of the mutable container of type MutableInt.
If you would like to use BigDecimal instead for the container you could not use the collect() method as container.add(employee.getSalary()); would not change the container because BigDecimal it is immutable.
(Apart from this BigDecimal::new would not work as BigDecimal has no empty constructor)
The normal reduction is meant to combine two immutable values such as int, double, etc. and produce a new one; it’s an immutable reduction. In contrast, the collect method is designed to mutate a container to accumulate the result it’s supposed to produce.
To illustrate the problem, let's suppose you want to achieve Collectors.toList() using a simple reduction like
List<Integer> numbers = stream.reduce(
new ArrayList<Integer>(),
(List<Integer> l, Integer e) -> {
l.add(e);
return l;
},
(List<Integer> l1, List<Integer> l2) -> {
l1.addAll(l2);
return l1;
});
This is the equivalent of Collectors.toList(). However, in this case you mutate the List<Integer>. As we know the ArrayList is not thread-safe, nor is safe to add/remove values from it while iterating so you will either get concurrent exception or ArrayIndexOutOfBoundsException or any kind of exception (especially when run in parallel) when you update the list or the combiner tries to merge the lists because you are mutating the list by accumulating (adding) the integers to it. If you want to make this thread-safe you need to pass a new list each time which would impair performance.
In contrast, the Collectors.toList() works in a similar fashion. However, it guarantees thread safety when you accumulate the values into the list. From the documentation for the collect method:
Performs a mutable reduction operation on the elements of this stream using a Collector. If the stream is parallel, and the Collector is concurrent, and either
the stream is unordered or the collector is unordered, then a
concurrent reduction will be performed. When executed in parallel, multiple intermediate results may be instantiated, populated, and merged so as to maintain isolation of mutable data structures. Therefore, even when executed in parallel with non-thread-safe data structures (such as ArrayList), no additional synchronization is needed for a parallel reduction.
So to answer your question:
When would you use collect() vs reduce()?
if you have immutable values such as ints, doubles, Strings then normal reduction works just fine. However, if you have to reduce your values into say a List (mutable data structure) then you need to use mutable reduction with the collect method.
Let the stream be a <- b <- c <- d
In reduction,
you will have ((a # b) # c) # d
where # is that interesting operation that you would like to do.
In collection,
your collector will have some kind of collecting structure K.
K consumes a.
K then consumes b.
K then consumes c.
K then consumes d.
At the end, you ask K what the final result is.
K then gives it to you.
They are very different in the potential memory footprint during the runtime. While collect() collects and puts all data into the collection, reduce() explicitly asks you to specify how to reduce the data that made it through the stream.
For example, if you want to read some data from a file, process it, and put it into some database, you might end up with java stream code similar to this:
streamDataFromFile(file)
.map(data -> processData(data))
.map(result -> database.save(result))
.collect(Collectors.toList());
In this case, we use collect() to force java to stream data through and make it save the result into the database. Without collect() the data is never read and never stored.
This code happily generates a java.lang.OutOfMemoryError: Java heap space runtime error, if the file size is large enough or the heap size is low enough. The obvious reason is that it tries to stack all the data that made it through the stream (and, in fact, has already been stored in the database) into the resulting collection and this blows up the heap.
However, if you replace collect() with reduce() -- it won't be a problem anymore as the latter will reduce and discard all the data that made it through.
In the presented example, just replace collect() with something with reduce:
.reduce(0L, (aLong, result) -> aLong, (aLong1, aLong2) -> aLong1);
You do not need even to care to make the calculation depend on the result as Java is not a pure FP (functional programming) language and cannot optimize out the data that is not being used at the bottom of the stream because of the possible side-effects.
Here is the code example
List<Integer> list = Arrays.asList(1,2,3,4,5,6,7);
int sum = list.stream().reduce((x,y) -> {
System.out.println(String.format("x=%d,y=%d",x,y));
return (x + y);
}).get();
System.out.println(sum);
Here is the execute result:
x=1,y=2
x=3,y=3
x=6,y=4
x=10,y=5
x=15,y=6
x=21,y=7
28
Reduce function handle two parameters, the first parameter is the previous return value int the stream, the second parameter is the current
calculate value in the stream, it sum the first value and current value as the first value in next caculation.
According to the docs
The reducing() collectors are most useful when used in a multi-level reduction, downstream of groupingBy or partitioningBy. To perform a simple reduction on a stream, use Stream.reduce(BinaryOperator) instead.
So basically you'd use reducing() only when forced within a collect.
Here's another example:
For example, given a stream of Person, to calculate the longest last name
of residents in each city:
Comparator<String> byLength = Comparator.comparing(String::length);
Map<String, String> longestLastNameByCity
= personList.stream().collect(groupingBy(Person::getCity,
reducing("", Person::getLastName, BinaryOperator.maxBy(byLength))));
According to this tutorial reduce is sometimes less efficient
The reduce operation always returns a new value. However, the accumulator function also returns a new value every time it processes an element of a stream. Suppose that you want to reduce the elements of a stream to a more complex object, such as a collection. This might hinder the performance of your application. If your reduce operation involves adding elements to a collection, then every time your accumulator function processes an element, it creates a new collection that includes the element, which is inefficient. It would be more efficient for you to update an existing collection instead. You can do this with the Stream.collect method, which the next section describes...
So the identity is "re-used" in a reduce scenario, so slightly more efficient to go with .reduce if possible.
There is a very good reason to always prefer collect() vs the reduce() method. Using collect() is much more performant, as explained here:
Java 8 tutorial
*A mutable reduction operation(such as Stream.collect()) collects the stream elements in a mutable result container(collection) as it processes them.
Mutable reduction operations provide much improved performance when compared to an immutable reduction operation(such as Stream.reduce()).
This is due to the fact that the collection holding the result at each step of reduction is mutable for a Collector and can be used again in the next step.
Stream.reduce() operation, on the other hand, uses immutable result containers and as a result needs to instantiate a new instance of the container at every intermediate step of reduction which degrades performance.*
Related
I have the following for loop:
List<Player> players = new ArrayList<>();
for (Team team : teams) {
ArrayList<TeamPlayer> teamPlayers = team.getTeamPlayers();
for (teamPlayer player : teamPlayers) {
players.add(new Player(player.getName, player.getPosition());
}
}
and I'm trying to convert it to a Stream:
List<Player> players = teams.forEach(t -> t.getTeamPlayers()
.forEach(p -> players.add(new Player(p.getName(), p.getPosition())))
);
But I'm getting a compilation error:
variable 'players' might not have been initialized
Why is this happening? Maybe there's an alternative way to create the stream, I was thinking of using flatMap but not sure how to apply it.
First of all, you need to understand that Streams don't act like Loops.
Hence, don't try to mimic a loop. Examine the tools offered by the API. Operation forEach() is there for special cases when you need to perform side-effects, not in order to accumulate elements from the stream into a Collection.
Note: with teams.forEach() you're not actually using a stream, but method Iterable.forEach() which is available with every implementation of Iterable.
To perform reduction on streams, we have several specialized operations like collect, reduce, etc. (for more information refer to the API documentation - Reduction).
collect() operation is meant to perform mutable reduction. You can use to collect the data into a list by providing built-in Collector Collectors.toList() as an argument. And since Java 16 operation toList() was introduced into API, which is implemented on top of the toArray() operation and performs better than namesake collector (therefore it's a preferred option if your JDK version allows you to use it).
I was thinking of using flatMap but not sure how to apply it.
Operation flatMap() is meant to perform one-to-many transformations. It expects a Function which takes a stream element and generates a Stream of the resulting type, elements of the generated stream become a replacement for the initial element.
Note: that general approach to writing streams to use as fewer operations as possible (because one of the main advantages that Functional programming brings to Java is simplicity). For that reason, applying flatMap() when a stream element produces a Collection in a single step is idiomatic, since it's sorter than performing map().flatMap() in two steps.
That's how implementation might look like:
List<Team> teams = List.of();
List<Player> players = teams.stream() // Stream<Team>
.flatMap(team -> team.getTeamPlayers().stream()) // Stream<Player>
.map(player -> new Player(player.getName(), player.getPosition()))
.toList(); // for Java 16+ or collect(Collectors.toList())
This is basically the answer of Alexander Ivanchenko, but with method reference.
final var players = teams.stream()
.map(Team::getTeamPlayers)
.flatMap(Collection::stream)
.map(p -> new Player(p.getName(), p.getPosition()))
.toList();
If your Player class has a factory method like (depending on the relation between Player and TeamPlayer:
public static Player fromTeamPlayer(final TeamPlayer teamPlayer) {
return new Player(teamPlayer.getName(), teamPlayer.getPosition());
}
You could further reduce it to:
final var players = teams.stream()
.map(Team::getTeamPlayers)
.flatMap(Collection::stream)
.map(Player::fromTeamPlayer)
.toList();
I am in the progress of learning through Java 8 lambda expressions and would like to ask about the following piece of Java code relating to the peek method in the function interface that I have come across.
On execution of the program on IDE, it gives no output. I was expecting it would give 2, 4, 6.
import java.util.Arrays;
import java.util.List;
public class Test_Q3 {
public Test_Q3() {
}
public static void main(String[] args) {
List<Integer> values = Arrays.asList(1, 2, 3);
values.stream()
.map(n -> n * 2)
.peek(System.out::print)
.count();
}
}
I assume you are running this under Java 9? You are not altering the SIZED property of the stream, so there is no need to execute either map or peek at all.
In other words all you care is about count as the final result, but in the meanwhile you do not alter the initial size of the List in any way (via filter for example or distinct) This is an optimization done in the Streams.
Btw, even if you add a dummy filter this will show what you expect:
values.stream ()
.map(n -> n*2)
.peek(System.out::print)
.filter(x -> true)
.count();
Here's some relevant quotes from the Javadoc of Stream interface:
A stream implementation is permitted significant latitude in optimizing the computation of the result. For example, a stream implementation is free to elide operations (or entire stages) from a stream pipeline -- and therefore elide invocation of behavioral parameters -- if it can prove that it would not affect the result of the computation. This means that side-effects of behavioral parameters may not always be executed and should not be relied upon, unless otherwise specified (such as by the terminal operations forEach and forEachOrdered). (For a specific example of such an optimization, see the API note documented on the count() operation. For more detail, see the side-effects section of the stream package documentation.)
And more specifically from the Javadoc of count() method:
API Note:
An implementation may choose to not execute the stream pipeline (either sequentially or in parallel) if it is capable of computing the count directly from the stream source. In such cases no source elements will be traversed and no intermediate operations will be evaluated. Behavioral parameters with side-effects, which are strongly discouraged except for harmless cases such as debugging, may be affected. For example, consider the following stream:
List<String> l = Arrays.asList("A", "B", "C", "D");
long count = l.stream().peek(System.out::println).count();
The number of elements covered by the stream source, a List, is known and the intermediate operation, peek, does not inject into or remove elements from the stream (as may be the case for flatMap or filter operations). Thus the count is the size of the List and there is no need to execute the pipeline and, as a side-effect, print out the list elements.
These quotes only appear on the Javadoc of Java 9, so it must be a new optimization.
I've encountered a situation that I though possible to handle using the Stream API but I simply cannot figure out a proper solution.
The case is the following : I have a stream of elements sorted by an identifier field. There are several elements with the same value for this identifier, and I need to deduplicate them based on conditions on other fields. Conceptually, it can be seen as a reduce operation on several chunks of the stream yielding to a stream of the same type.
For now, the only solution I manage to come with, is to collect the stream based on the common identifier to obtain something like Map<Id, List<Elem>> and then use this map's stream to apply my deduplication rules and go on. The problem (and why I won't use this solution) is that collect is a terminal operation, re-streaming after it means that I will iterate over my elements twice.
UPDATE
Consider the following class :
public static class Item {
private final int _id;
private final double _price;
public Item(final int id, final double price) {
_id = id;
_price = price;
}
public int id() {
return _id;
}
public double price() {
return _price;
}
}
And the following stream :
final Stream<Item> items = Stream.<Item>builder()
.add(new Item(1, 4))
.add(new Item(1, 6))
.add(new Item(1, 3))
.add(new Item(2, 5))
.add(new Item(2, 1))
.add(new Item(3, 5))
.build();
After the required operation, if the rule of deduplication is "with the highest price", the stream should only contains Item(1, 6), Item(2, 5) and Item(3, 5).
If I do this imperatively, I can consume my items while they have the same id, backing them up in a temporary collection, and deduplicate this collection when encountering an item with a different id.
If I use collect to first group the items by id, I will consume all the data at once before moving to the next operation, and I need to avoid that.
For most cases of that kind, a temporary storage, like a Map, is inevitable. After all, it’s the map’s efficient lookup algorithm that allows to identify the group each element belongs to. Also, it’s possible that the first group contains the first and the very last element of the source stream and the only way to find out whether this is the case, is iterating the entire source stream. This might not be true for your special case of pre-sorted data, but the API doesn’t provide a way to exploit this for a grouping operation. And it wouldn’t play nicely with parallel Stream support, if it existed.
But consider the groupingBy collector accepting a downstream Collector which allows you to reduce the groups to their final result in-place. If it is a true reduction, you can use, e.g. reducing as downstream collector. This allows you to collect the elements into a Map<Id, Reduced> rather than Map<Id, List<Elem>>, so you don’t collect into Lists that have to be reduced afterwards.
For any similar case, if you can describe the follow-up operation as a Collector, its processing will indeed start right when encountering the first element of a group. Note that there are other combining Collectors like mapping and collectingAndThen. Java 9 will also add filtering and flatMapping, so you can express a lot of typical Stream operations in form of a downstream collector. For convenience, this collector combines a mapping step with a follow-up reduction step.
Further processing of the groups can only be done after the full completion of the grouping, by accessing Map.values(). If the final result is supposed to be a Collection, it’s not necessary to stream over it again, as the existing collection operations are sufficient, e.g. you can use new ArrayList<>(map.values()) if you need a List rather than an unspecific Collection.
If your concern is that the operation should not be performed until the caller commences a terminal operation on the final Stream, you can use an operation like this:
public Stream<ResultType> stream() {
return StreamSupport.stream(() -> items.stream()
.collect(Collectors.groupingBy(classificationFunc,
Collectors.reducing(id, mappingFunc, reductionFunc)))
.values().spliterator(),
Spliterator.SIZED, false);
}
I haven't tested this, but using the StreamEx library, you should be able to collapse() adjacent elements like this:
items.collapse((a, b) -> a.id() == b.id(), (a, b) -> a.price() < b.price() ? b : a)
I have a method in a data Structure that I wish to use to pass various collectors and apply them to my object.
The following is the Method -->
public <R> R applyCollector(String key, Collector a)
{
this.key = key;
this.a = a;
R result = (R) this.stateList.stream().
map(state -> state.getKey(key)).collect(a);
return result;
}
The above method basically takes in a "key" and a Collector that it applies over the values got by key.
This is the way I'm using it -->
Collector stringToListCollector =
Collectors.toList();
List<String> values =
myObject.applyCollector("key",
stringToListCollector);
This works fine for simple things like getting count, average etc.
But, what if I wish to send something more complex, like a Nested Collector.
For example, say my "key" returns me a String, which is actually an IP or even an Integer.
What I'd like to do is to send a collector that first Converts my String to Integer by doing a "integer::parseInt" and then doing the toList.
Right now I have to first retrieve a list named values (defined above). And then do a values.stream().Map(Integer::ParseString).collect(Collectors::averagingInt).
I might need to do this operation multiple times, I have two options.
Make the ToList and Map and Collect as a function. and call it. This
beats my purpose of lambda.
Write or nest existing Collectors to directly do that for me. This
option looks neater to me because if I can do it correctly, I'll be
able to do everything in One Pass instead of the 2 passes it takes
me now, as well as maybe save the Memory I need to first create a
list.
How do I do this? Write a Collector that does -->
Gets an object, and runs Integer::ParseInt upon it and then do an
Average Operation.
For your example it would look like
applyCollector("key", mapping(Integer::parseInt, averagingInt(i -> i))
Collectors can be composed to some extend:
Collectors.mapping executes function before collecting
Collectors.collectingAndThen executes function after collecting
additionally some collectors accept downstream collectors i.e groupingBy
I'm still trying to fully grasp working with the Stream package in Java 8 and was hoping for some help.
I have a class, described below, instances of which I receive in a list as part of a database call.
class VisitSummary {
String source;
DateTime timestamp;
Integer errorCount;
Integer trafficCount;
//Other fields
}
To generate some possibly useful information about this, I have a class VisitSummaryBySource which holds the sum total of all visits (for a given timeframe):
class VisitSummaryBySource {
String sourceName;
Integer recordCount;
Integer errorCount;
}
I was hoping to construct a List<VisitSummaryBySource> collection which as the name sounds, holds the list of VisitSummaryBySource objects containing the sum total of records and errors encountered, for each different source.
Is there a way I can achieve this using streams in a single operation? Or do I need to necessarily break this down into multiple operations? The best I could come up with is:
Map<String, Integer> recordsBySrc = data.parallelStream().collect(Collectors.groupingBy(VisitSummaryBySource::getSource,
Collectors.summingInt(VisitSummaryBySource::getRecordCount)));
and to calculate the errors
Map<String, Integer> errorsBySrc = data.parallelStream().collect(Collectors.groupingBy(VisitSummaryBySource::getSource,
Collectors.summingInt(VisitSummaryBySource::getErrorCount)));
and merging the two maps to come up with the list I'm looking for.
You're on the right track. The uses of Collectors.summingInt are examples of downstream collectors of the outer groupingBy collector. This operation extracts one of the integer values from each VisitSummaryBySource instance in the same group, and sums them. This is essentially a reduction over integers.
The problem, as you note, is that you can extract/reduce only one of the integer values, so you have to perform a second pass to extract/reduce the other integer values.
The key is to consider reduction not over the individual integer values but over the entire VisitSummaryBySource object. Reduction takes a BinaryOperator, which takes two instances of the type in question and combines them into one. Here's how to do that, by adding a static method to VisitSummaryBySource:
static VisitSummaryBySource merge(VisitSummaryBySource a,
VisitSummaryBySource b) {
assert a.getSource().equals(b.getSource());
return new VisitSummaryBySource(a.getSource(),
a.getRecordCount() + b.getRecordCount(),
a.getErrorCount() + b.getErrorCount());
}
Note that we're not actually merging the source names. Since this reduction is only performed within a group, where the source names are the same, we assert that we can only merge two instances whose names are the same. We also assume the obvious constructor taking a name, record count, and error count, and call that to create the merged object, containing the sums of the counts.
Now our stream looks like this:
Map<String, Optional<VisitSummaryBySource>> map =
data.stream()
.collect(groupingBy(VisitSummaryBySource::getSource,
reducing(VisitSummaryBySource::merge)));
Note that this reduction produces map values of type Optional<VisitSummaryBySource>. This is somewhat odd; we'll deal with it below. We could avoid the Optional by using another form of the reducing collector that takes an identity value. This is possible but somewhat nonsensical, as there's no good value to use for the source name of the identity. (We could use something like the empty string, but we'd have to abandon our assertion that we merge only objects whose source names are equal.)
We don't really care about the map; it only needs to be kept around long enough to reduce the VisitSummaryBySource instances. Once that's done, we can just pull out the map values using values() and throw away the map.
We can also turn this back into a stream and unwrap the Optional by mapping them through Optional::get. This is safe, because a value never ends up in the map unless there's at least one member of the group.
Finally, we collect the results into a list.
The final code looks like this:
List<VisitSummaryBySource> output =
data.stream()
.collect(groupingBy(VisitSummaryBySource::getSource,
reducing(VisitSummaryBySource::merge)))
.values().stream()
.map(Optional::get)
.collect(toList());