There is a problem on my server where it became a bottle neck due to a specific problem to solve resolving a List<List<SomeObject>> into a List<SomeObject>. The CPU of the server spiked above normal means.
DataStructure is:
Object:
List<SomeObject> childList;
Trying to make a List<Object> flatmapped to List<SomeObject> in the most computationally efficient way.
If parentList = List<Object>:
I Tried:
parentList.stream().flatMap(child -> child.getChildList().stream()).collect(Collectors.toList())
Also tried:
List<Object> all = new ArrayList<>();
parentList.forEach(child -> all.addAll(child.getChildList()))
Any other suggestions? These seem to be similar in computation but pretty high due to copying underneath the hood.
This may be more efficient since it eliminates creating multiple streams via flatMap. MapMulti was introduced in Java 16. It takes the streamed argument and a consumer which puts something on the stream, in this case each list's object.
List<List<Object>> lists = new ArrayList<>(
List.of(List.of("1", "2", "3"),
List.of("4", "5", "6", "7"),
List.of("8", "9")));
List<Object> list = lists.stream().mapMulti(
(lst, consumer) -> lst.forEach(consumer))
.toList();
System.out.print(list);
prints
[1, 2, 3, 4, 5, 6, 7, 8, 9]
Do we know more about which List implementation is used?
I would try to init the resulting list with the correct expected size.
This avoids unnecessary copying.
This assumes that the size of the lists can be retrieved fast.
int expectedSize = parentList.stream()
.mapToInt(entry -> entry.getChildList().size())
.sum();
List<SomeObject> result = new ArrayList<>(expectedSize);
for (var entry : parentList) {
result.addAll(entry.getChildList());
}
In java 8
List<Object> listOne = new ArrayList<>();
List<Object> listTwo = new ArrayList<>();
List<Object> listThree = new ArrayList<>();
...
Stream.of(...) concatenate many lists
List<Object> newList = Stream.of(listOne,listTwo,listThree).flatMap(Collection::stream).collect(Collectors.toList());
In Java 16+
List<Object> newList=Stream.concat(Stream.concat(listOne, listTwo), listThree).toList();
Being an ETL (“Extract Transform and Load”) process, Streams processes collections of data using multiple threads of execution at each stage of processing.
One way to make the flat mapping more computationally efficient is to use a for loop instead of the stream API or forEach method. The for loop would iterate over the parent list, and for each element, it would add the child list to the flat list. This avoids the overhead of creating streams and using the collect method. Additionally, using an ArrayList to store the flat list instead of a LinkedList can also improve performance since it has a more efficient implementation of the addAll method.
List<SomeObject> flatList = new ArrayList<>();
for (Object o : parentList) {
flatList.addAll(o.getChildList());
Another way would be to use an iterator. Iterator is an interface for traversing a collection and it's more efficient than forEach or for loop.
List<SomeObject> flatList - new ArrayList<>();
Iterator<Object> iterator = parentList.iterator();
while(iterator.hasNext()){
Object o = iterator.next():
flatList.addAll(o.getChildList()):
}
You could also use the concat method for List, which concatenates two lists in an efficient way and results in a new list.
List<SomeObject> flatList = new ArrayList<>()
for (Object o : parentList){
flatList.concat(o.getChildList());
}
THERE ARE SERVERAL RESOURCES THAT YOU CAN USE FOR ADDITIONAL READING ON THIS TOPIC. HERE ARE A FEW THAT I WOULD RECOMMEND.
https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/util/List.html
https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/util/ArrayList.html
https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/util/Iterator.html
https://www.oreilly.com/library/view/java-performance-the/9781449358652/
https://www.tutorialspoint.com/java_data_structure_algorithms/index.htm
1) What is the difference between
for (MyObject myObject : sessionBean.firstList()) {
secondList.add(new MyObject(myObject.getSeverity(), myObject.getSummary(), myObject.getDetail()));
}
and
for (MyObject myObject : sessionBean.firstList()) {
secondList.add(facesMessage);
}
considering that i later want to clear sessionBean.firstList but want to retain the values in secondList.
2) Is there a more readable way of doing it in java 8?
You can do it in single statement in a readable way:
secondList.addAll(sessionBean.firstList())
When you use new you will allocate additional memory, otherwise you are mapping the reference of the existing objects. Even if you want to clear the sessionBean.firstList later point of time, if you have a valid reference for that objects in another collection,it will not be collected by garbage collector. So I could not see any valid scenario to compel to use the new operator.
1) The difference is that in the first case the secondList will contain references to new objects with same fields as in the firstList. And in the second case secondList will contain references to the same objects as in the firstList. So if you modify one of your firstList items, the corresponding item in the secondList will be modified too.
2) You can use the following example to create a list with copies:
List<MyObject> secondList = sessionBean.firstList()
.stream()
.map(first -> new MyObject(first.getSeverity(), first.getSummary(), first.getDetail()))
.collect(Collectors.toList());
Or, if you implement a clone method in your MyObject class
public MyObject clone() {
return new MyObject(severity, summary, detail);
}
you can simply write:
List<MyObject> secondList = sessionBean.firstList()
.stream()
.map(MyObject::clone)
.collect(Collectors.toList());
Or if you just want to copy references, use
secondList.addAll(sessionBean.firstList())
List<Object1> list1 = userVo.getMstCardTypeVOs();
Object1 object1 = new Object1();
object1.setId(1);
object1.setName("Test");
-- More fields which are not matched with object2
list1.add(object1);
object1 = new Object1();
object1.setId(2);
object1.setName("Test1");
-- More fields which are not matched with object1
list1.add(object1)
List<Object2> list2 = mapMenuProgramRepo.findAll();
Object2 object2 = new Object2();
object2.setId(1);
object2.setName("Test");
list1.add(object2);
object2 = new Object2();
object2.setId(2);
object2.setName("Test1");
list1.add(object2);
What I need to check same id exist in list1 with reference from list2?
I have used below code:
for (Object1 object1 : obj1) {
for (Object2 obj2: obj2) {
if (object2.getId().equals(object1.getId())) {
// removed entry from the list1
// We can removed by using iterator instead of list
}
}
}
What are the better way and optimized way using jdk8 or 7?
// removed entry from the list1
If you need to remove from one of lists than you can use following code
list1.removeIf(val1 -> list2.stream()
.filter(val2 -> val2.getId().equals(val1.getId()).findFirst().orElse(null) != null)
Although using stream like other answers looks cool, imho it may not be optimal, especially when there is a lot of values in list2.
Instead of using stream to verify if a value exists in list2, which gives complexity of O(M*N) (M,N for number of values in list1 and list2), I would recommend doing things like:
Set<> excludeIds = list2.stream().map(Object2::getId).collect(Collectors.toSet());
list1.removeIf(v -> excludeIds.contains(v.getId()));
Easier to understand, and run faster.
With a HashSet, the complexity of the logic reduce to O(M+N)
You can do this easily with JDK8 streams as shown below by using noneMatch() method:
list1.removeIf(value1 ->
list2.stream().anyMatch(value2 ->
(value2.getId().equals(value1.getId))));//equal ids then remove it from list1
If you have to solve the problem without using streams, then you need to use Iterator (look here for example on remove) and then loop through the list1 and list2 and remove the matched elements from list1
Also, I suggest never create any class names (even example classes) with Object1 or Object2, etc.. as it is very confusing for other developers(i.e., it is not easy to understand the code with class names as Object1, Object2).
In java suppose I have 2 lists
List<Object1> list1
List<Object2> list2
object1.getName(); returns a String
object2.getName(); return a String
is there any way to compare the names and get a difference of the two list
those 2 objects are defined in the 3rd party library, and I can't override the equals and compareto methods
I am in favour of googles Guava or commons collections library
but the Sets.symmetricDifference(Set1, Set2) ask for 2 to be passed in,
even i juse Sets.newHashSet(lis1) and Sets.newHashSet(lis2) to create two sets
but still they have difference type of objects in the sets.
or in commons CollectionUtils.disjunction(lis1, list2) the lists still has to contain the same object type
without doing 2 expensive for loops, is there any other way?
First, we'll build two maps, one for each list, mapping names to objects. Then we iterate over the differences between the key sets, processing whichever kind of object had that name. The maps let us avoid scanning through the list looking for the object with that name. (In using Map rather than Multimap, I'm relying on the asker's comment on another answer that within each list, names are unique. If you're still using Java 7, replace the method reference with a Function implementation.)
Map<String, Object1> map1 = Maps.uniqueIndex(list1, Object1::getName);
Map<String, Object2> map2 = Maps.uniqueIndex(list2, Object1::getName);
for (String name : Sets.difference(map1.keySet(), map2.keySet()))
processObject1(map1.get(name));
for (String name : Sets.difference(map2.keySet(), map1.keySet()))
processObject2(map2.get(name));
If all you want to do is build lists or sets of the objects in exactly one list, processObject1 and processObject2 can just add the objects to collections.
uniqueIndex's iteration order is that of the input iterable, and difference returns a SetView with the same iteration order as its first argument, so you can process objects in the order they appeared in the input lists, if that order is relevant to your problem.
Java 8 streams provide basically the same functionality:
Map<String, Object1> map1 = list1.stream().collect(Collectors.toMap(Function.identity(), Object1::getName));
Map<String, Object2> map2 = list2.stream().collect(Collectors.toMap(Function.identity(), Object2::getName));
map1.keySet().stream().filter(n -> !map2.keySet().contains(n)).map(map1::get).forEachOrdered(o1 -> processObject1(o1));
map2.keySet().stream().filter(n -> !map1.keySet().contains(n)).map(map2::get).forEachOrdered(o2 -> processObject1(o2));
Again, you can replace the forEachOrdered call with collect(Collectors.toList()) if you just want to collect the objects.
First you will have to transfor your lists to String based lists:
private static final class FromObject1ToName implements Function<Object1, String> {
#Override
public String apply(Object1 input) {
return input.name;
}
}
The same transformation has to be done for Object2
Then transform the input list:
Collection<String> transformed = Collections2.transform(list1, new FromObject1ToName());
//list1 is a List on Object1
Then create the multiset:
Multiset<String> multiset1 = HashMultiset.create();
multiset1.addAll(transformed);
Then simply do :
Multisets.difference(multiset1, multiset2) // multiset1 is from Object1 and multiset2 is from Object2
This will give you the difference and how many times it differes
If you need to know just the differences, then do the same transform, then load the Collection of strings in a Set adn then do Sets.symmetricDifference
Using Guava, try this. It works for me ->
Multisets.difference(multiset1,multiset2);
How to convert ArrayList to Multiset.
List x = new ArrayList();
x.add(3);.....
Multiset newX = HashMultiset.create();
newX.addAll(x);
What is the difference between
List<Integer> list1 = new ArrayList<Integer>(Arrays.asList(ia)); // Copy
List<Integer> list2 = Arrays.asList(ia);
, where ia is an array of integers?
I came to know that some operations are not allowed in list2. Why is it so?
How is it stored in memory (references / copy)?
When I shuffle the lists, list1 doesn't affect the original array, but list2 does. But still list2 is somewhat confusing.
How does ArrayList being upcasted to list differ from creating a new ArrayList?
list1 differs from (1)
ArrayList<Integer> list1 = new ArrayList<Integer>(Arrays.asList(ia));
First, let's see what this does:
Arrays.asList(ia)
It takes an array ia and creates a wrapper that implements List<Integer>, which makes the original array available as a list. Nothing is copied and all, only a single wrapper object is created. Operations on the list wrapper are propagated to the original array. This means that if you shuffle the list wrapper, the original array is shuffled as well, if you overwrite an element, it gets overwritten in the original array, etc. Of course, some List operations aren't allowed on the wrapper, like adding or removing elements from the list, you can only read or overwrite the elements.
Note that the list wrapper doesn't extend ArrayList - it's a different kind of object. ArrayLists have their own, internal array, in which they store their elements, and are able to resize the internal arrays etc. The wrapper doesn't have its own internal array, it only propagates operations to the array given to it.
On the other hand, if you subsequently create a new array as
new ArrayList<Integer>(Arrays.asList(ia))
then you create new ArrayList, which is a full, independent copy of the original one. Although here you create the wrapper using Arrays.asList as well, it is used only during the construction of the new ArrayList and is garbage-collected afterwards. The structure of this new ArrayList is completely independent of the original array. It contains the same elements (both the original array and this new ArrayList reference the same integers in memory), but it creates a new, internal array, that holds the references. So when you shuffle it, add, remove elements etc., the original array is unchanged.
Well, this is because ArrayList resulting from Arrays.asList() is not of the type java.util.ArrayList.
Arrays.asList() creates an ArrayList of type java.util.Arrays$ArrayList which does not extend java.util.ArrayList, but only extends java.util.AbstractList.
List<Integer> list1 = new ArrayList<Integer>(Arrays.asList(ia)); //copy
In this case, list1 is of type ArrayList.
List<Integer> list2 = Arrays.asList(ia);
Here, the list is returned as a List view, meaning it has only the methods attached to that interface. Hence why some methods are not allowed on list2.
ArrayList<Integer> list1 = new ArrayList<Integer>(Arrays.asList(ia));
Here, you are creating a new ArrayList. You're simply passing it a value in the constructor. This is not an example of casting. In casting, it might look more like this:
ArrayList list1 = (ArrayList)Arrays.asList(ia);
First of all, the Arrays class is a utility class which contains a number of utility methods to operate on Arrays (thanks to the Arrays class. Otherwise, we would have needed to create our own methods to act on Array objects)
asList() method:
asList method is one of the utility methods of Array class, it is a static method that's why we can call this method by its class name (like Arrays.asList(T...a) )
Now here is the twist. Please note that this method doesn't create new ArrayList object. It just returns a List reference to an existing Array object (so now after using asList method, two references to existing Array object gets created)
and this is the reason. All methods that operate on List object, may not work on this Array object using the List reference. Like
for example, Arrays size is fixed in length, hence you obviously can not add or remove elements from Array object using this List reference (like list.add(10) or list.remove(10);. Else it will throw UnsupportedOperationException).
any change you are doing using a list reference will be reflected in the exiting Arrays object (as you are operating on an existing Array object by using a list reference)
In the first case, you are creating a new Arraylist object (in the second case, only a reference to existing Array object is created, but not a new ArrayList object), so now there are two different objects. One is the Array object and another is the ArrayList object and there isn't any connection between them (so changes in one object will not be reflected/affected in another object (that is, in case 2, Array and Arraylist are two different objects)
Case 1:
Integer [] ia = {1,2,3,4};
System.out.println("Array : "+Arrays.toString(ia));
List<Integer> list1 = new ArrayList<Integer>(Arrays.asList(ia)); // new ArrayList object is created , no connection between existing Array Object
list1.add(5);
list1.add(6);
list1.remove(0);
list1.remove(0);
System.out.println("list1: " + list1);
System.out.println("Array: " + Arrays.toString(ia));
Case 2:
Integer [] ia = {1,2,3,4};
System.out.println("Array: " + Arrays.toString(ia));
List<Integer> list2 = Arrays.asList(ia); // Creates only a (new) List reference to the existing Array object (and NOT a new ArrayList Object)
// list2.add(5); // It will throw java.lang.UnsupportedOperationException - invalid operation (as Array size is fixed)
list2.set(0,10); // Making changes in the existing Array object using the List reference - valid
list2.set(1,11);
ia[2]=12; // Making changes in the existing Array object using the Array reference - valid
System.out.println("list2: " + list2);
System.out.println("Array: " + Arrays.toString(ia));
An explanation with documentation references would be better for someone looking for answer.
1. java.util.Arrays
This is a utility class with bunch of static methods to operate on given array
asList is one such static method that takes input array and returns an object of java.util.Arrays.ArrayList which is a static nested class that extends AbstractList<E> which in turn implements List interface.
So Arrays.asList(inarray) returns a List wrapper around the input array, but this wrapper is java.util.Arrays.ArrayList and not java.util.ArrayList and it refers to the same array, so adding more elements to the List wrapped array would affect the original one too and also we cannot change the length.
2. java.util.ArrayList
ArrayList has a bunch of overloaded constructors
public ArrayList() - // Returns arraylist with default capacity 10
public ArrayList(Collection<? extends E> c)
public ArrayList(int initialCapacity)
So when we pass the Arrays.asList returned object, i.e., List(AbstractList) to the second constructor above, it will create a new dynamic array (this array size increases as we add more elements than its capacity and also the new elements will not affect the original array) shallow copying the original array (shallow copy means it copies over the references only and does not create a new set of same objects as in original array)
String names[] = new String[]{"Avinash","Amol","John","Peter"};
java.util.List<String> namesList = Arrays.asList(names);
or
String names[] = new String[]{"Avinash","Amol","John","Peter"};
java.util.List<String> temp = Arrays.asList(names);
The above statement adds the wrapper on the input array. So the methods like add and remove will not be applicable on the list reference object 'namesList'.
If you try to add an element in the existing array/list then you will get "Exception in thread "main" java.lang.UnsupportedOperationException".
The above operation is readonly or viewonly.
We can not perform add or remove operation in list object.
But
String names[] = new String[]{"Avinash","Amol","John","Peter"};
java.util.ArrayList<String> list1 = new ArrayList<>(Arrays.asList(names));
or
String names[] = new String[]{"Avinash","Amol","John","Peter"};
java.util.List<String> listObject = Arrays.asList(names);
java.util.ArrayList<String> list1 = new ArrayList<>(listObject);
In the above statement you have created a concrete instance of an ArrayList class and passed a list as a parameter.
In this case, methods add and remove will work properly as both methods are from ArrayList class, so here we won't get any UnSupportedOperationException.
Changes made in the Arraylist object (method add or remove an element in/from an arraylist) will get not reflect in to the original java.util.List object.
String names[] = new String[] {
"Avinash",
"Amol",
"John",
"Peter"
};
java.util.List < String > listObject = Arrays.asList(names);
java.util.ArrayList < String > list1 = new ArrayList < > (listObject);
for (String string: list1) {
System.out.print(" " + string);
}
list1.add("Alex"); // Added without any exception
list1.remove("Avinash"); // Added without any exception will not make any changes in original list in this case temp object.
for (String string: list1) {
System.out.print(" " + string);
}
String existingNames[] = new String[] {
"Avinash",
"Amol",
"John",
"Peter"
};
java.util.List < String > namesList = Arrays.asList(names);
namesList.add("Bob"); // UnsupportedOperationException occur
namesList.remove("Avinash"); // UnsupportedOperationException
Note that, in Java 8, 'ia' above must be Integer[] and not int[]. Arrays.asList() of an int array returns a list with a single element. When using the OP's code snippet, the compiler will catch the issue, but some methods (e.g., Collections.shuffle()) will silently fail to do what you expect.
Many people have answered the mechanical details already, but it's worth noting:
This is a poor design choice, by Java.
Java's asList method is documented as "Returns a fixed-size list...". If you take its result and call (say) the .add method, it throws an UnsupportedOperationException. This is unintuitive behavior! If a method says it returns a List, the standard expectation is that it returns an object which supports the methods of interface List. A developer shouldn't have to memorize which of the umpteen util.List methods create Lists that don't actually support all the List methods.
If they had named the method asImmutableList, it would make sense. Or if they just had the method return an actual List (and copy the backing array), it would make sense. They decided to favor both runtime-performance and short names, at the expense of violating both the principle of least astonishment and the good object-oriented practice of avoiding UnsupportedOperationExceptions.
(Also, the designers might have made a interface ImmutableList, to avoid a plethora of UnsupportedOperationExceptions.)
package com.copy;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
public class CopyArray {
public static void main(String[] args) {
List<Integer> list1, list2 = null;
Integer[] intarr = { 3, 4, 2, 1 };
list1 = new ArrayList<Integer>(Arrays.asList(intarr));
list1.add(30);
list2 = Arrays.asList(intarr);
// list2.add(40); Here, we can't modify the existing list,because it's a wrapper
System.out.println("List1");
Iterator<Integer> itr1 = list1.iterator();
while (itr1.hasNext()) {
System.out.println(itr1.next());
}
System.out.println("List2");
Iterator<Integer> itr2 = list2.iterator();
while (itr2.hasNext()) {
System.out.println(itr2.next());
}
}
}
Arrays.asList()
This method returns its own implementation of List. It takes an array as an argument and builds methods and attributes on top of it, since it is not copying any data from an array but using the original array this causes alteration in original array when you modify list returned by the Arrays.asList() method.
On the other hand, ArrayList(Arrays.asList());
is a constructor of ArrayList class which takes a list as argument and returns an ArrayList that is independent of list, i.e., Arrays.asList() in this case passed as an argument.
That is why you see these results.
1.List<Integer> list1 = new ArrayList<Integer>(Arrays.asList(ia)); //copy
2.List<Integer> list2 = Arrays.asList(ia);
In line 2, Arrays.asList(ia) returns a List reference of inner class object defined within Arrays, which is also called ArrayList but is private and only extends AbstractList. This means what returned from Arrays.asList(ia) is a class object different from what you get from new ArrayList<Integer>.
You cannot use some operations to line 2 because the inner private class within Arrays does not provide those methods.
Take a look at this link and see what you can do with the private inner class:
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/Arrays.java#Arrays.ArrayList
Line 1 creates a new ArrayList object copying elements from what you get from line 2. So you can do whatever you want since java.util.ArrayList provides all those methods.
In response to some comments asking questions about the behaviour of Arrays.asList() since Java 8:
int[] arr1 = {1,2,3};
/*
Arrays are objects in Java, internally int[] will be represented by
an Integer Array object which when printed on console shall output
a pattern such as
[I#address for 1-dim int array,
[[I#address for 2-dim int array,
[[F#address for 2-dim float array etc.
*/
System.out.println(Arrays.asList(arr1));
/*
The line below results in Compile time error as Arrays.asList(int[] array)
returns List<int[]>. The returned list contains only one element
and that is the int[] {1,2,3}
*/
// List<Integer> list1 = Arrays.asList(arr1);
/*
Arrays.asList(arr1) is Arrays$ArrayList object whose only element is int[] array
so the line below prints [[I#...], where [I#... is the array object.
*/
System.out.println(Arrays.asList(arr1));
/*
This prints [I#..., the actual array object stored as single element
in the Arrays$ArrayList object.
*/
System.out.println(Arrays.asList(arr1).get(0));
// prints the contents of array [1,2,3]
System.out.println(Arrays.toString(Arrays.asList(arr1).get(0)));
Integer[] arr2 = {1,2,3};
/*
Arrays.asList(arr) is Arrays$ArrayList object which is
a wrapper list object containing three elements 1,2,3.
Technically, it is pointing to the original Integer[] array
*/
List<Integer> list2 = Arrays.asList(arr2);
// prints the contents of list [1,2,3]
System.out.println(list2);
Summary of the difference -
When a list is created without using the new, the operator Arrays.asList() method returns a wrapper which means:
you can perform an add/update operation.
the changes done in the original array will be reflected to List as well and vice versa.