We Keep Coding

Problem witn putIfAbsent() when modifying a stream

Please help me to figure out what is wrong with the method bellow and how can I solve it. The method takes a stream of Person object and Map with String value (a task name) as a key and an int value (a mark). The purpose of the method is to check whether a stream contains the particular tasks from allHistoryTasks variable and if does apply to this Map .putIfAbsentmethod(taskName, 0) to ensure that all the tasks are present in the Map (the purpose is to calculate an average mark later).
When I run the test the UnsupportedOperationException error apears. When I comment the lines from the if statement and to forEach (lines 1, 2, 3, 4) test runs well. I'm new to Java and already spent several days on this issue but still can't solve it. Please tell me what is wrong here.
private Set<String> allHistoryTasks = Set.of("Phalanxing", "Shieldwalling", "Tercioing", "Wedging");
private String[] historyTasks = allHistoryTasks.toArray(new String[0]);
public Map<Person, Map<String, Integer>> addHistoryIfPresent(Stream<CourseResult> stream) {
return stream.collect(Collectors.toMap(
CourseResult::getPerson,
x -> {
if (allHistoryTasks.containsAll(x.getTaskResults().keySet())) //1
IntStream.range(0, allHistoryTasks.size()) //2
.parallel() //3
.forEach(i -> x.getTaskResults().putIfAbsent(historyTasks[i], 0)); //4
return x.getTaskResults();
}
));
}
custom classes & thread report

The x -> {} block is the 'value mapper'. It is supposed to turn an element of your stream into the value for a given map.
You have a stream of CourseResult objects, and want a Map<Person, Map<String, Integer>>, so this function turns a CourseResultobject into aMap<String, Integer>`.
You do this by mutating things and that is a biiig nono. Your stream functions should not have any side-effects. Fortunately, the author of CourseResult is way ahead of you and blocked you from making this error. You are calling .getTaskResults() on your course result object and then trying to modify it. You can't do that, as the getTaskResults() method returns a map that cannot be modified.
Presumably, you want to clone that map, and fix the clone. How do you do that? Well, you tell me, the API isn't clear. You could simply make a new ImmutableMap.builder(), loop through whatever you want to loop through, and so on. From your code it's not quite clear what end map you do want.
Note also that you're using powers without knowing what you're doing - you have a parallel stream and are then forEaching through it, mutating the same variable, which you absolutely cannot do: This results in bugs where the result of an operation depends on an evil coin flip, in the sense that it can work fine today even if you rerun the tests a million times, and fail tomorrow. Separately, using parallel() for such talks is borderline crazy - assuming the underlying stream impl actually parallelizes (.parallel() is a hint, not a demand), it would just slow everything waaay down. allHistoryTasks is tiny. This isn't what parallelism would be for.

This might be the answer to your question. Set.of method won't return the mutable set. So you need to declare a mutrable set like this to avoid this problem.
private Set<String> allHistoryTasks = new HashSet<>(Arrays.asList("Phalanxing", "Shieldwalling", "Tercioing", "Wedging"));
private String[] historyTasks = allHistoryTasks.toArray(new String[0]);
public Map<Person, Map<String, Integer>> addHistoryIfPresent(Stream<CourseResult> stream) {
return stream.collect(Collectors.toMap(
CourseResult::getPerson,
x -> {
if (allHistoryTasks.containsAll(x.getTaskResults().keySet())) //1
IntStream.range(0, allHistoryTasks.size()) //2
.parallel() //3
.forEach(i -> x.getTaskResults().putIfAbsent(historyTasks[i], 0)); //4
return x.getTaskResults();
}
));
}

Aggregate values and convert into single type within the same Java stream

I have a class with a collection of Seed elements. One of the method's return type of Seed is Optional<Pair<Boolean, String>>.
I'm trying to loop over all seeds, find if any boolean value is true and at the same time, create a set with all the String values. For instance, my input is in the form Optional<Pair<Boolean, String>>, the output should be Optional<Signal> where Signal is like:
class Signal {
public boolean exposure;
public Set<String> alarms;
// constructor and getters (can add anything to this class, it's just a bag)
}
This is what I currently have that works:
// Seed::hadExposure yields Optional<Pair<Boolean, String>> where Pair have key/value or left/right
public Optional<Signal> withExposure() {
if (seeds.stream().map(Seed::hadExposure).flatMap(Optional::stream).findAny().isEmpty()) {
return Optional.empty();
}
final var exposure = seeds.stream()
.map(Seed::hadExposure)
.flatMap(Optional::stream)
.anyMatch(Pair::getLeft);
final var alarms = seeds.stream()
.map(Seed::hadExposure)
.flatMap(Optional::stream)
.map(Pair::getRight)
.filter(Objects::nonNull)
.collect(Collectors.toSet());
return Optional.of(new Signal(exposure, alarms));
}
Now I have time to make it better because Seed::hadExposure could become and expensive call, so I was trying to see if I could make all of this with only one pass. I've tried (some suggestions from previous questions) with reduce, using collectors (Collectors.collectingAndThen, Collectors.partitioningBy, etc.), but nothing so far.

It's possible to do this in a single stream() expression using map to convert the non-empty exposure to a Signal and then a reduce to combine the signals:
Signal signal = exposures.stream()
.map(exposure ->
new Signal(
exposure.getLeft(),
exposure.getRight() == null
? Collections.emptySet()
: Collections.singleton(exposure.getRight())))
.reduce(
new Signal(false, new HashSet<>()),
(leftSig, rightSig) -> {
HashSet<String> alarms = new HashSet<>();
alarms.addAll(leftSig.alarms);
alarms.addAll(rightSig.alarms);
return new Signal(
leftSig.exposure || rightSig.exposure, alarms);
});
However, if you have a lot of alarms it would be expensive because it creates a new Set and adds the new alarms to the accumulated alarms for each exposure in the input.
In a language that was designed from the ground-up to support functional programming, like Scala or Haskell, you'd have a Set data type that would let you efficiently create a new set that's identical to an existing set but with an added element, so there'd be no efficiency worries:
filteredSeeds.foldLeft((false, Set[String]())) { (result, exposure) =>
(result._1 || exposure.getLeft, result._2 + exposure.getRight)
}
But Java doesn't come with anything like that out of the box.
You could create just a single Set for the result and mutate it in your stream's reduce expression, but some would regard that as poor style because you'd be mixing a functional paradigm (map/reduce over a stream) with a procedural one (mutating a set).
Personally, in Java, I'd just ditch the functional approach and use a for loop in this case. It'll be less code, more efficient, and IMO clearer.
If you have enough space to store an intermediate result, you could do something like:
List<Pair<Boolean, String>> exposures =
seeds.stream()
.map(Seed::hadExposure)
.flatMap(Optional::stream)
.collect(Collectors.toList());
Then you'd only be calling the expensive Seed::hadExposure method once per item in the input list.

Remake list with some condition

There are two entities:
class GiftCertificate {
Long id;
List<Tag> tags;
}
class Tag {
Long id;
String name;
}
There is a list
List<GiftCertificate>
which contains, for example, the following data:
<1, [1, "Tag1"]>, <2, null>, <1, [2, "Tag2"]>. (It does not contain a set of tags, but only one tag or does not have it at all).
I need to do so that in the result it was this:
<1, {[1," Tag1 "], [2," Tag2 "]}>, <2, null>. I mean, add to the set of the first object a tag from the third GiftCertificate and at the same time delete the 3rd one. I would like to get at least some ideas on how to do this. it would be nice to use stream.

Probably not the most effective way, but it might help
private List<GiftCertificate> joinCertificates(List<GiftCertificate> giftCertificates) {
return giftCertificates.stream()
.collect(Collectors.groupingBy(GiftCertificate::getId))
.entrySet().stream()
.map(entry -> new GiftCertificate(entry.getKey(), joinTags(entry.getValue()))).collect(Collectors.toList());
}
private List<Tag> joinTags(List<GiftCertificate> giftCertificates) {
return giftCertificates.stream()
.flatMap(giftCertificate -> Optional.ofNullable(giftCertificate.getTags()).stream().flatMap(Collection::stream))
.collect(Collectors.toList());
}

You can do what you want with streams and with the help of a dedicated custom constructor and a couple of helper methods in GiftCertificate. Here's the constructor:
public GiftCertificate(GiftCertificate another) {
this.id = another.id;
this.tags = new ArrayList<>(another.tags);
}
This just works as a copy constructor. We're creating a new list of tags, so that if the list of tags of either one of the GiftCertificate instances is modified, the other one won't. (This is just basic OO concepts: encapsulation).
Then, in order to add another GiftCertificate's tags to this GiftCertificate's list of tags, you could add the following method to GiftCertificate:
public GiftCertificate addTagsFrom(GiftCertificate another) {
tags.addAll(another.tags);
return this;
}
And also, a helper method that returns whether the list of tags is empty or not will come in very handy:
public boolean hasTags() {
return tags != null && !tags.isEmpty();
}
Finally, with these three simple methods in place, we're ready to use all the power of streams to solve the problem in an elegant way:
Collection<GiftCertificate> result = certificates.stream()
.filter(GiftCertificate::hasTags) // keep only gift certificates with tags
.collect(Collectors.toMap(
GiftCertificate::getId, // group by id
GiftCertificate::new, // use our dedicated constructor
GiftCertificate::addTagsFrom)) // merge the tags here
.values();
This uses Collectors.toMap to create a map that groups gift certificates by id, merging the tags. Then, we keep the values of the map.
Here's the equivalent solution, without streams:
Map<Long, GiftCertificate> map = new LinkedHashMap<>(); // preserves insertion order
certificates.forEach(cert -> {
if (cert.hasTags()) {
map.merge(
cert.getId(),
new GiftCertificate(cert),
GiftCertificate::addTagsFrom);
}
});
Collection<GiftCertificate> result = map.values();
And here's a variant with a slight performance improvement:
Map<Long, GiftCertificate> map = new LinkedHashMap<>(); // preserves insertion order
certificates.forEach(cert -> {
if (cert.hasTags()) {
map.computeIfAbsent(
cert.getId(),
k -> new GiftCertificate(k)) // or GitCertificate::new
.addTagsFrom(cert);
}
});
Collection<GiftCertificate> result = map.values();
This solution requires the following constructor:
public GiftCertificate(Long id) {
this.id = id;
this.tags = new ArrayList<>();
}
The advantage of this approach is that new GiftCertificate instances will be created only if there's no other entry in the map with the same id.

Java 9 introduced flatMapping collector that is particularly well-suited for problems like this. Break the task into two steps. First, build a map of gift certificate IDs to list of tags and then assemble a new list of GiftCertificate objects:
import static java.util.stream.Collectors.flatMapping;
import static java.util.stream.Collectors.groupingBy;
import static java.util.stream.Collectors.toList;
......
Map<Long, List<Tag>> gcIdToTags = gcs.stream()
.collect(groupingBy(
GiftCertificate::getId,
flatMapping(
gc -> gc.getTags() == null ? Stream.empty() : gc.getTags().stream(),
toList()
)
));
List<GiftCertificate> r = gcIdToTags.entrySet().stream()
.map(e -> new GiftCertificate(e.getKey(), e.getValue()))
.collect(toList());
This assumes that GiftCertificate has a constructor that accepts Long id and List<Tag> tags
Note that this code deviates from your requirements by creating an empty list instead of null in case there are no tags for a gift certificate id. Using null instead of an empty list is just a very lousy design and forces you to pollute your code with null checks everywhere.
The first argument to flatMapping can also be written as gc -> Stream.ofNullable(gc.getTags()).flatMap(List::stream) if you find that more readable.

Performance of algorithms sorting list entries into a map

Given a list in which each entry is a object that looks like
class Entry {
public String id;
public Object value;
}
Multiple entries could have the same id. I need a map where I can access all values that belong to a certain id:
Map<String, List<Object>> map;
My algorithm to achieve this:
for (Entry entry : listOfEntries) {
List<Object> listOfValues;
if (map.contains(entry.id)) {
listOfValues = map.get(entry.id);
} else {
listOfValues = new List<Object>();
map.put(entry.id, listOfValues);
}
listOfValues.add(entry.value);
}
Simply: I transform a list that looks like
ID | VALUE
---+------------
a | foo
a | bar
b | foobar
To a map that looks like
a--+- foo
'- bar
b---- foobar
As you can see, contains is called for each entry of the source list. That's why I wonder if I could improve my algorithm, if I pre-sort the source list and then do this:
List<Object> listOfValues = new List<Object>();
String prevId = null;
for (Entry entry : listOfEntries) {
if (prevId != null && prevId != entry.id) {
map.put(prevId, listOfValues);
listOfValues = new List<Object>();
}
listOfValues.add(entry.value);
prevId = entry.id;
}
if (prevId != null) map.put(prevId, listOfValues);
The second solution has the advantage that I don't need to call map.contains() for every entry but the disadvantage that I have to sort before. Futhermore the first algorithm is easier to implement and less error prone, since you have to add some code after the actual loop.
Therefore my question is: Which method has better performance?
The examples are written in Java pseudo code but the actual question applies to other programming languages as well.

If you have a hash map and a very large amount of entries then inserting items one by one will be faster than sorting and inserting them list by list (O(n) vs O(N log N)). If you use a tree based map than the complexity is the same for both approaches.
However, I really doubt you have a sufficiently large amount of entries so memory access patterns, and how fast compare and hash functions are come into effect. You have 2 options: ignore it since the difference is not going to be significant or benchmark both options and see which one is working better on your system. If you don't have millions of entries I would ignore the issue and go with whatever is easier to understand.

Don't presort. Even fast sorting algorithms like quicksort take, on average, O(n log n) for n items. Afterwards, you still need O(n) to walk the list. contains on a (hash) map takes constant time (checkout this question), don't worry about it. Walk the list in linear time and use contains.

Would like to offer another solution using streams
import static java.util.stream.Collectors.groupingBy;
import static java.util.stream.Collectors.mapping;
import static java.util.stream.Collectors.toList;
Map<String, List<Object>> map = listOfValues.stream()
.collect(groupingBy(entry -> entry.id, mapping(entry -> entry.value, toList())));
This code is more declarative - it only specifies that List should be transformed into Map.
Then it is a library responsibility to actually perform transformation in efficient way.

How can we have entry-level locking in HashMap?

Since, statckoverflow does not allow add more thing to your question in the original question (you can only add comment, not code) I am asking a sequential question to my original question here:
Can we use Synchronized for each entry instead of ConcurrentHashMap?
The problem is very simple, and I don't know why for such a simple problem that probably many people have encountered before me I should spend this much time :/
The problem is: I have a hashmap, I want when one thread is working on one of the entries of the hashMap, no any other thread access that object, and I don't want to lock the whole hashMap.
I know that java provides ConcurrentHashMap, but ConcurrentHashMap does not solve the problem, when you want to do thing more complex than simple put and get. Even newly added functions (in Java 8) like merge is not enough for complex scenarios.
For example:
Suppose I want a hash map that maps strings to ArrayLists. Then for example suppose I want to do this:
For key k, if there is any entry, add newString to its ArrayList, but if there is no entry for k, create the entry for k such that its ArrayList has newString.
I was thinking I can do it as follows:
ArrayList<String> tm =new ArrayList<String>();
tm.add(newString);
Object result = map.putIfAbsent(k, tm);
if (result != null)
{
map.get(k).add(newString);
}
But it does not work, why? suppose putIfAbset return something other than null, then it means that map already has an entry with key k, so I will try to add newString to the ArrayList of the already existing entry, but right before adding, another thread may remove the entry, and then I will get NullPointerException!
So, I found it very difficult to code such things properly.
But I was thinking that if I can simply lock that entry, life will be wonderful!
In my previous post I suggested something very simple that in fact eliminates the need for concurrentHashMap, and provide entry-level locking but some said that is not true because Long is not immutable ... that I didn't get it well.
Now, I implemented and tested it, it looks good to me, but I don't know why other more experienced developers here told me it is not thread-safe :(
This is the exact code that I tested:
MainThread:
import java.util.HashMap;
public class mainThread {
public static HashMap<String, Long> map = new HashMap<String, Long>();
public static void main (String args[])
{
map.put("k1", new Long(32));
synchronized(map.get("k1"))
{
Thread t = new Thread(new threadA());
t.start();
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
ThreadA:
public class ThreadA implements Runnable {
#Override
public void run() {
mainThread.map.put("k2", new Long(21));
System.out.println(mainThread.map.get("k2"));
synchronized (mainThread.map.get("k1")) {
System.out.println("Insdie synchronized of threadA");
}
}
}
It works fine! It prints 21, and after 5 seconds, that mainThread release the lock of map.get("k1"), it prints "Insdie synchronized of threadA"
So, why using this simple approach we cannot provide entry-level locking?! why concurrency should be that much complicated Lol (just kidding)

First of all, there is no standard map implementation that I am aware of that provides entry level locking.
But I think you can avoid the need for that. For example
UPDATE ... corrected mistake
ArrayList<String> tm = new ArrayList<String>();
ArrayList<String> old = map.putIfAbsent(k, tm);
if (old != null) {
tm = old;
}
synchronized (tm) {
// can now add / remove entries and this will appear as an atomic
// actions to other threads that are using `synchronized` to
// access or update the list
tm.add(string1);
tm.add(string2);
}
Yes it is possible that another thread will update the list in the hashmap entry between this thread (possibly) inserting it, and this thread locking it. However, that doesn't matter. The (corrected) putIfAbsent and the test that follows ensures that everyone will use and lock the same list.
(Assumption: that all threads use this logic when inserting / updating an entry.)
Atomically removing the list if it becomes empty is difficult, but I would argue that it is usually unnecessary to do that.
UPDATE 2
There is a better way:
ArrayList<String> tm = map.computeIfAbsent(k, ArrayList::new);
synchronized (tm) {
...
}
(Thanks Stuart)
UPDATE 3
We can do it with merger too.
Maybe, yes. Something like this:
ArrayList<String> tm = new ArrayList<String>;
tm.add(...);
...
map.merge(key, tm, (oldV, newV) -> {oldV.addAll(newV); return oldV});
The downside is that you are double-handling all the elements of tm; i.e. adding to 2 separate lists (one of which you throw way).
But you could also do this:
map.merge(key, tm, (oldV, newV) -> {
oldV.removeAll(newV);
return oldV.size() == 0 ? null : oldV}
);
The thing that concerns me is that the javadoc does not state explicitly that the value oldV will be locked while this is happening. It says:
"The entire method invocation is performed atomically. Some attempted update operations on this map by other threads may be blocked while computation is in progress ..."
... but it does not explicitly state that there is mutual exclusion on the value while this is happening. (For instance, mixing this approach with putIfAbsent / computeIfAbsent and an explicit synchronized block would most likely be hazardous. The locking would most likely be on different objects.)

Well, the first huge problem is that you don't even attempt to do any locking for the put calls. Those aren't automatically threadsafe for a regular HashMap. You seem to be under the impression that separate HashMap entries are completely independent automatically, but HashMaps don't work that way.
Even if you fix the put problem (probably requiring ConcurrentHashMap or a whole-map lock anyway), the parts you actually are locking for aren't locking safely.
Say thread 1 puts the entry "k1": 1, and thread 2 tries to get("k1"). What will thread 2 see?
Well, thread 2 doesn't even try to acquire any locks until the get call is already done. The get call is completely unprotected! Without any happens-before relation between the put and the get, the get call might not see the entry, or it might see the entry, or it might see the map in an inconsistent intermediate state and crash horribly.
Synchronizing on the result of the get call is synchronizing far too late.

I think I have finally found the solution using merge function. I provide an example, I will edit this post to make it easier for others to read, but I just post now to have your feedback.
Here is the example of a ConcurrentHashMap that has ConcurrentHashMaps as its values (23 and 1 are just two random value for sake of example):
Long intialValue = new Long(3);
Long addedValue = new Long(10);
Long removingValue = new Long (5);
ConcurrentHashMap<Integer, ConcurrentHashMap<Integer, Long>> map = new ConcurrentHashMap<>();
//Initialization....
ConcurrentHashMap<Integer, Long> i = new ConcurrentHashMap<Integer, Long>();
i.put(1, intialValue);
map.put(23, i);
//......
//addition
ConcurrentHashMap<Integer, Long> c = new ConcurrentHashMap<Integer, Long>();
c.put(1, addedValue);
map.merge(23, c, (oldHashMap, newHashMap) -> {
oldHashMap.merge (1, c.get(1), (oldV, newV) -> {
if (oldV < newV) return newV; else return oldV;
});
return oldHashMap;
});
//removal
// we want to remove entry 1 from the inner HashMap if its value is less than 2, and if the entry is empty remove the entry from the outer HashMap
ConcurrentHashMap<Integer, Long> r = new ConcurrentHashMap<Integer, Long>();
r.put(1, removingValue);
map.merge (23, r, (oldHashMap, newHashMap) -> {
oldHashMap.merge(1, newHashMap.get(1), (oldV, newV) -> {if (oldV < newV) return newV; else return oldV;});
return oldHashMap;
});
map.remove(23, r);
if (map.containsKey(23))
{
System.out.println("Map contains key 23");
if (map.get(23).containsKey(1))
{
System.out.println("The value for <23,1> is " + map.get(23).get(1));
}
}
This is what the code does:
initialization: first creates the map and puts another map into it for key 23 which has value initialValue for key 1.
addition: Then checks, 1) if for key 23, there is no value, it puts a map that has value addedValue for key 1, otherwise 2) if key 23 has already a value, it checks its value if the value has a value less than addedValue, it overwrites it with the addedValue, otherwise it leaves it alone.
removal: Finally, it checks, if for key 23, and for key 1 in the value for 23, the value is less than removingValue, it removes that, and if the hashMap of key 23 is empty after this removal, it removes key 23 from the main map.
I tested this code. So for example:
for 3, 10, 5, the final value for <23,1> is 10.
for 20, 10, 11, the final value is 20.
for 3, 10, 11, the final value is nothing,because entry 23 is
removed.
I hope it is thread-safe as I just used merge method. One disadvantage of this code is that I am adding something to map and then remove it, just because ConcurrentHashMap does not have a method for remove similar to merge. I wish I had this method:
map.remove (keyToRemove, condition)