We Keep Coding

Filtering/Removing from a Nested List of Objects using Streams in Java

Say that we have a 3-dimensional List of Objects.
class OneDObject {
int id;
List<Integer> list;
OneDObject(int id, List<Integer>list) { // Constructor }
// Getters and Setters
}
class TwoDObject {
int id;
List<OneDObject> list;
TwoDObject(int id, List<OneDObject> list) { // Constructor }
// Getters and Setters
}
var l1 = List.of(1,2,4);
var l2 = List.of(2,4,6);
var obj1d1 = new OneDObject(1, l1);
var obj1d2 = new OneDObject(2, l2);
var l3 = List.of(obj1d1, obj1d2);
var l4 = List.of(obj1d1);
var obj2d1 = new TwoDObject(3, l3);
var obj2d2 = new TwoDObject(4, l4);
var l5 = List.of(obj2d1, obj2d2); // 3-d list
Say that I want to filter "l5" such that if any element in the inner most list is an odd number then the entire list should be deleted, and if that makes the 2nd level list as empty, then that should be deleted in return.
So, for the given example, before filtering if it is:
[[[1,2,4],[2,4,6]], [[1,2,4]]]
After filtering, it should be:
[[[2,4,6]]]
How can I do this using Streams in Java?

Since you need your lists to be updated, in the below solution I am using removeIf method of the List to remove any elements which does not meet the necessary criteria. So for removeIf to work, the list should not be immutable. So replace the var list = List.of(...) code with var list = new ArrayList<>(List.of(...));
(Note: Null checks have been ignored as well.)
Now, this problem could be split into components:
Predicate to identify if a list has any odd elements.
Predicate<OneDObject> hasOdd = obj-> obj.getList().stream().anyMatch(i -> i % 2 != 0);
Predicate to remove objects from 2d list, which has odd elements in its 1d list.
Predicate<TwoDObject> validate2d = obj -> {
// remove any 1d list that has atleast one odd number.
obj.getList().removeIf(hasOdd);
// check if there are any valid 1d lists
return obj.getList().isEmpty();
};
Now apply the predicate to the final list:
l5.removeIf(validate2d); // l5 will now contain only the 2d object having [2,4,6] list

Here's the final code (in Java, but I think it should almost be interchangeable with Kotlin)
List<TwoDObject> l6 = l5.stream()
.peek(twoDObject -> {
List<OneDObject> filteredOneDObjectList = twoDObject.getList()
.stream()
.filter(oneDObject -> oneDObject.getList()
.stream()
.noneMatch(i -> i % 2 == 1))
.toList();
twoDObject.setList(filteredOneDObjectList);
})
.filter(twoDObject -> twoDObject.getList().size() > 0)
.toList();
First we go through every twoDObject by calling Stream#peek, then stream its list and filter out every oneDObject, which contains an odd number. Then the list is saved back into the current twoDObject.
In the end we filter out all empty twoDObjects.
Note that Stream#peek should normally only be used for the purpose of debugging and not mutating the stream elements.
In this case it could also be replaced with
List<TwoDObject> l6 = l5.stream()
.map(twoDObject -> {
...
return twoDObject;
})
...

Parameterized test using custom objects

I am trying to write a Parameterized test for my add method for Binary Search tree
I wrote the test method:
void add() {
student5 = new Student("student5", LocalDate.of(2000,5,12), "none");
studentSet2.add(student5);
assertAll(
()->assertThat(studentSet2).contains(student5),
()->assertThat(studentSet2).hasSize(1),
()->assertThat(studentSet2.iterator().next()).isEqualTo(student5),
()->assertThat(studentSet2.toArray(new Student[1])).containsOnly(student5)
);
This is my current test method but i want to transform it using parameterized test.But when i started to learn about it ,i found that he can take only strings and primitiv types.
Can i write something for my test method to take a "Student" object as parameter?

You want create a lot of synthetic cases (or read from somewhere...), for example
static Stream<Arguments> testAddToAListSource() {
final List<String> baseData = List.of("one", "two", "three");
// construct cases using a base list and adding null or a new element
return IntStream.range(0, baseData.size() + 1)
.mapToObj(n -> baseData.stream().limit(n).collect(toList()))
.flatMap(xs -> Stream.of(null, "extra", "superExtra")
.map(x -> arguments(new ArrayList<>(xs), x)));
}
this method create pairs of a list of strings and a new string to add.
Suppose you want to verify that this element is correctly added to the list checking the following invariants
#ParameterizedTest
#MethodSource("testAddToAListSource")
void testAddToAList(List<String> caseList, String newElement) {
// count elements
final long currentSize = caseList.size();
// count how many times the element is in list
final long currentTimes = caseList.stream().filter(e -> Objects.equals(e, newElement)).count();
// add
caseList.add(newElement);
// count elements
final long newSize = caseList.size();
// count how many times the element is in list
final long newTimes = caseList.stream().filter(e -> Objects.equals(e, newElement)).count();
assertEquals(newSize, currentSize + 1);
assertEquals(newTimes, currentTimes + 1);
}
with output
in your specific case I don't know what type is studentSet2 but probably you could change your add signature to
#ParameterizedTest
#MethodSource("testAdd")
void add(StudentSet studentSet2, Student student5) {
...
and create a provider like
static Stream<Arguments> testAdd() {
...

Correct order of elements of Cartesian product of ArrayLists

I tried to generate Cartesian product of unknown number of ArrayLists (of fixed type) based on this answer: Cartesian product of an arbitrary number of sets. But I have found something strange. The cartesian products is always given in reverse order. For example, if A and B are two Lists, B's elements are given first and A's elements are given second in cartesian pair. What could be possible reason? How to fix that? Original answerer says, ordering does not matter in Cartesian product. But I think ordering is the main thing while making cartesian products especially when each set represents coordinates of plane.
Modified Code:
private static Set<ArrayList<Double>> cartesianProduct(ArrayList<ArrayList<Double>> sets) {
if (sets.size() < 2)
throw new IllegalArgumentException(
"Can't have a product of fewer than two sets (got " +
sets.size() + ")");
return _cartesianProduct(0, sets);
}
private static Set<ArrayList<Double>> _cartesianProduct(int index, ArrayList<ArrayList<Double>> sets) {
Set<ArrayList<Double>> ret = new HashSet<>();
if (index == sets.size()) {
ret.add(new ArrayList<>());
} else {
for (Double obj : sets.get(index)) {
for (ArrayList<Double> set : _cartesianProduct(index + 1, sets)) {
set.add(obj);
ret.add(set);
}
}
}
return ret;
}
Output:
ArrayList<Double> l1 = new ArrayList<>(Arrays.asList(1.0, 2.0));
ArrayList<Double> l2 = new ArrayList<>(Arrays.asList(4.0, 5.0));
ArrayList<ArrayList<Double>> l = new ArrayList<>(Arrays.asList(l1, l2));
Set<ArrayList<Double>> a = cartesianProduct(l);
// a = [[4.0, 1.0], [4.0, 2.0], [5.0, 1.0], [5.0, 2.0]]

This method constructs the cartesian product in reverse because it creates the product "inside out" - when it's returning out of the recursion.
Printed out the value returned by each level in the recursion and you'll see how it happens.
The second level of recursion works on list B and returns [[4], [5]].
The first level of recursion takes [[4], [5]] and uses the list.add method to add items from the list A. This method adds items to the end of the list so the result is [[4, 1], [5, 1], [4, 2], [5, 2]].
How to fix it?
A quick fix is inserting items to the front, instead of to the back. Instead of set.add(obj) use:
set.add(0, obj);
Another option is to reverse the order of iteration so that the second level of recursion uses list A, and the first level uses list B. The initial call to start the recursion would be made from sets.size() and it should count down instead of up:
return _cartesianProduct(sets.size() - 1, sets);
...
for (ArrayList<Double> set : _cartesianProduct(index - 1, sets)) {
Yet another option is changing the recursion so that the product is built on the way down the recursion - "outside in" - instead of on the way out. This is the approach taken in another answer to the question you link to: https://stackoverflow.com/a/9496234/318758

This happens because of recursion. Index is initially 0, so at the line for (ArrayList<Double> set : _cartesianProduct(index + 1, sets)) {, your code calls cartesianProduct again with index=1. Again it reaches that line, and calls cartesianProduct with index=2. When it is at index=2, it reaches its base case and returns a set with an empty ArrayList.
Then it goes back to the stackframe where index=1 (remember,obj is 4.0 because sets.get(1) is the ArrayList containing 4 and 6). It adds all the doubles in sets.get(index) (here it is 4.0 and 6.0) to their own ArrayLists in ret. Then it reaches the end of the foreach loop and returns the set, which now has 2 ArrayLists, one containing a 4.0 and the other 6.0.
The same happens at index=0, so the first list(or set)'s elements are added after the second list's elements. That's why you get reverse results.
To fix this, you could decrement index every time, going from sets.size() to 0 instead of the other way around. To reverse it, you can also simply call Collections.reverse() on every set inside the result.
//Fix by decrementing index
private static Set<ArrayList<Double>> cartesianProduct(ArrayList<ArrayList<Double>> sets) {
if (sets.size() < 2)
throw new IllegalArgumentException(
"Can't have a product of fewer than two sets (got " + sets.size() + ")");
//Be sure to start at the end of 'sets' so you can go down by one
return cartesianProduct(sets.size() - 1, sets);
}
private static Set<ArrayList<Double>> cartesianProduct(int index, ArrayList<ArrayList<Double>> sets) {
Set<ArrayList<Double>> ret = new HashSet<>();
//Counting to 0 instead of to the end of the sets ArrayList
if (index < 0) {
ret.add(new ArrayList<>());
} else {
for (Double obj : sets.get(index)) {
for (ArrayList<Double> set : cartesianProduct(index - 1, sets)) {
set.add(obj);
ret.add(set);
}
}
}
return ret;
}
//Alternative answer using Collections.reverse
private static Set<ArrayList<Double>> cartesianProduct(ArrayList<ArrayList<Double>> sets) {
if (sets.size() < 2)
throw new IllegalArgumentException(
"Can't have a product of fewer than two sets (got " + sets.size() + ")");
//This basically goes through the set of sets and reverses each ArrayList
return cartesianProduct(0, sets).stream().map(Collections::reverse).collect(Collectors.toSet());
}

If order matters, you can rewrite your code as follows:
ArrayList<Double> l1 = new ArrayList<>(Arrays.asList(1.0, 2.0, 3.0));
ArrayList<Double> l2 = new ArrayList<>(Arrays.asList(4.0, 5.0, 6.0));
List<List<Double>> cartesianProduct = Stream.of(l1, l2)
// represent each list element as a singleton list
.map(list -> list.stream().map(Collections::singletonList)
// List<List<Double>>
.collect(Collectors.toList()))
// intermediate output
//[[1.0], [2.0], [3.0]]
//[[4.0], [5.0], [6.0]]
.peek(System.out::println)
// summation of pairs of inner lists
.reduce((list1, list2) -> list1.stream()
// combinations of inner lists
.flatMap(inner1 -> list2.stream()
// merge two inner lists into one
.map(inner2 -> Stream.of(inner1, inner2)
.flatMap(List::stream)
.collect(Collectors.toList())))
// list of combinations
.collect(Collectors.toList()))
// returns List<List<Double>>, otherwise an empty list
.orElse(Collections.emptyList());
// final output
cartesianProduct.forEach(System.out::println);
//[1.0, 4.0]
//[1.0, 5.0]
//[1.0, 6.0]
//[2.0, 4.0]
//[2.0, 5.0]
//[2.0, 6.0]
//[3.0, 4.0]
//[3.0, 5.0]
//[3.0, 6.0]
See also: Cartesian product of an arbitrary number of sets

How to cache and replay the items of a Supplier<Stream<T>>

Regarding a Supplier<Stream<T>> dataSrc I would like to cache the Stream items for further traversals of the same sequence of elements. In this case, assume that dataSrc always produces the same sequence (e.g. getting a Stream<Integer> with temperatures in Celsius of March (see example usage bellow)). Thus, option 1) is to first collect the Stream items, however it will waste one first traversal to add those items into a collection:
Supplier<Stream<T>> dataSrc = ...
List<T> cache = dataSrc.collect(toList()); // **Additional** traversal to collect items
cache.stream().reduce(…) // 1st traversal
cache.stream().reduce(…) // 2nd traversal
... // Nth traversals
I would like to avoid the additional traversal to collect items and the explicit cache variable and hide it inside the Supplier<> in such a way that on first traversal the items are implicitly cached and on further traversals the items are accessed from that cache. I think this is similar to the idea of the method cache() of Reactor Project for reactive streams.
Thus, I am outlining an alternative in the following cache() method implementation, although it already has two problems (at least): 1) the onClose is not called on traversal finish (and I cannot figure out any way of detecting the end of a traversal); 2) If the first traversal never ends then the cache will never be filled.
Supplier<Stream<T>> dataSrc = cache(...)
dataSrc.get().reduce(…) // 1st traversal
dataSrc.get().reduce(…) // 2nd traversal
... // Nth traversals
static <T> Supplier<Stream<T>> cache(Supplier<Stream<T>> dataSrc) {
final List<T> cache = new ArrayList<>();
final AtomicBoolean started = new AtomicBoolean();
final AtomicBoolean isCached = new AtomicBoolean();
return () -> {
if(isCached.get()) return cache.stream();
if(!started.getAndSet(true)) {
return dataSrc
.get()
.peek(cache::add)
.onClose(() -> isCached.set(true));
}
return dataSrc.get();
};
}
Question:
Is there any better approach to achieve an utility cache() function which returns a new Stream<T> that caches items on first Stream traversal (without an implicit additional traversal to collect first) and further Stream objects are created from that cache?
Usage Example:
Here I am getting a stream with the temperatures in March from the World Weather online API. To execute it you must include a dependency of AsyncHttpClient and a valid API key in given URI.
Pattern pat = Pattern.compile("\\n");
boolean [] isEven = {true};
CompletableFuture<Stream<Integer>> temps = asyncHttpClient()
.prepareGet("http://api.worldweatheronline.com/premium/v1/past-weather.ashx?q=37.017,-7.933&date=2018-03-01&enddate=2018-03-31&tp=24&format=csv&key=715b185b36034a4c879141841182802")
.execute()
.toCompletableFuture()
.thenApply(Response::getResponseBody)
.thenApply(pat::splitAsStream)
.thenApply(str -> str
.filter(w -> !w.startsWith("#")) // Filter comments
.skip(1) // Skip line: Not Available
.filter(l -> isEven[0] = !isEven[0]) // Filter Even line
.map(line -> line.substring(14, 16)) // Extract temperature in celcius
.map(Integer::parseInt)
);
Note that a CompletableFuture<Stream<Integer>> is functionally compliant with a Supplier<Stream<Integer>>. Although, the CompletableFuture caches the resulting stream it cannot be iterated twice.
Problem 1: The following code throws IllegalStateException: stream has already been operated upon or closed
out.println(temps.join().distinct().count());
out.println(temps.join().max(Integer::compare)); // throws IllegalStateException
Problem 2: Collecting it in a List will induce a first traversal and thus we will have 3 traversals, instead of 2:
CompletableFuture<List<Integer>> list = temps.thenApply(str -> str.collect(toList()));
out.println(list.join().stream().distinct().count()); // 2 traversals
out.println(list.join().stream().distinct().max(Integer::compare));// 1 traversal
Goal: Store items in cache on first traversal. Every time the stream retrieves an item it should store it in an internal cache that will be used on further traversals.
Supplier<Stream<Integer>> cache = Cache.of(temps::join);
out.println(temps.get().distinct().count()); // 1 traversal
out.println(temps.get().max(Integer::compare)); // 1 traversal form cache

I think the only way to detect the end of the Stream traversal is through its iterator() or spliterator(). Thus, maybe a better option to get a replayable Stream is to record its items from its iterator (done by Recorder class of the example bellow) and then implement a new Spliterator that reads the items previously recorded (done by cacheIterator()). In this solution I made the getOrAdvance() method of Recorder synchronized to guarantee that just one resulting stream will get a new item from the source.
So, Cache.of(dataSrc) creates a chain of:
dataSrc ----> Recorder ----> cacheIterator() ----> Stream
Side notes:
The resulting stream from the method Cache.of() permits limited parallelism. For better splitting support the cacheIterator() bellow should return a Spliterator implementation instead, such as AbstractList.RandomAccessSpliterator.
Although it is not a requirement, the Recorder/ cacheIterator() solution also works with infinite data sources that can be short-circuited later.
E.g. it can cache the items of the infinite stream nrs and prints the output bellow without, or with cache (i.e. nrsReplay):
Random rnd = new Random();
Supplier<Stream<String>> nrs = () -> Stream.generate(() -> rnd.nextInt(99)).map(Object::toString);
IntStream.range(1, 6).forEach(size -> out.println(nrs.get().limit(size).collect(joining(","))));
System.out.println();
Supplier<Stream<String>> nrsReplay = Cache.of(nrs);
IntStream.range(1, 6).forEach(size -> out.println(nrsReplay.get().limit(size).collect(joining(","))));
Output:
32
65,94
94,19,34
72,77,66,18
88,41,34,97,28
93
93,65
93,65,71
93,65,71,40
93,65,71,40,68
class Cache {
public static <T> Supplier<Stream<T>> of(Supplier<Stream<T>> dataSrc) {
final Spliterator<T> src = dataSrc.get().spliterator(); // !!!maybe it should be lazy and memorized!!!
final Recorder<T> rec = new Recorder<>(src);
return () -> {
// CacheIterator starts on index 0 and reads data from src or
// from an internal cache of Recorder.
Spliterator<T> iter = rec.cacheIterator();
return StreamSupport.stream(iter, false);
};
}
static class Recorder<T> {
final Spliterator<T> src;
final List<T> cache = new ArrayList<>();
final long estimateSize;
boolean hasNext = true;
public Recorder(Spliterator<T> src) {
this.src = src;
this.estimateSize = src.estimateSize();
}
public synchronized boolean getOrAdvance(
final int index,
Consumer<? super T> cons) {
if (index < cache.size()) {
// If it is in cache then just get if from the corresponding index.
cons.accept(cache.get(index));
return true;
} else if (hasNext)
// If not in cache then advance the src iterator
hasNext = src.tryAdvance(item -> {
cache.add(item);
cons.accept(item);
});
return hasNext;
}
public Spliterator<T> cacheIterator() {
return new Spliterators.AbstractSpliterator<T>(
estimateSize, src.characteristics()
) {
int index = 0;
public boolean tryAdvance(Consumer<? super T> cons) {
return getOrAdvance(index++, cons);
}
public Comparator<? super T> getComparator() {
return src.getComparator();
}
};
}
}
}

You can use Guava's Suppliers#memoize function to turn a given Supplier into a caching ("memoizing") one.
Turn your dataSrc Supplier<Stream<T>> into a Supplier<List<T>> that collects the stream
Wrap it with Suppliers#memoize
This would be your cache() method:
private static <T> Supplier<Stream<T>> cache(Supplier<Stream<T>> dataSrc) {
Supplier<List<T>> memoized = Suppliers.memoize(() -> dataSrc.get().collect(toList()));
return () -> memoized.get().stream();
}
(when mixing in Guava you might need to switch between Guava's version of c.g.c.b.Supplier, and java.util.Supplier, and they can easily be transformed back and forth, however in this case it's not even necessary)
Example
Assume a simple Integer stream that returns the first 5 natural numbers and reports computation to stdout:
private static Supplier<Stream<Integer>> getDataSrc() {
return () -> IntStream.generate(new IntSupplier() {
private int i = 0;
#Override
public int getAsInt() {
System.out.println("Computing next i: " + (i + 1));
return i += 1;
}
}).limit(5).boxed();
}
Then running the non-memoized version
Supplier<Stream<Integer>> dataSrc = getDataSrc();
System.out.println(dataSrc.get().collect(toList()));
System.out.println(dataSrc.get().collect(toList()));
yields
Computing next i: 1
Computing next i: 2
Computing next i: 3
Computing next i: 4
Computing next i: 5
[1, 2, 3, 4, 5]
Computing next i: 1
Computing next i: 2
Computing next i: 3
Computing next i: 4
Computing next i: 5
[1, 2, 3, 4, 5]
And running the memoized version
Supplier<Stream<Integer>> dataSrc = cached(getDataSrc());
System.out.println(dataSrc.get().collect(toList()));
System.out.println(dataSrc.get().collect(toList()));
yields
Computing next i: 1
Computing next i: 2
Computing next i: 3
Computing next i: 4
Computing next i: 5
[1, 2, 3, 4, 5]
[1, 2, 3, 4, 5]

If using the Reactor Project is an option, then you can simply convert the Supplier<Stream<T>> to a Flux<T>, which already provides the utility cache() method and henceforward use Flux<T> operations rather than Stream<T> operations.
Regarding the example of the original post, where temps is a CompletableFuture<Stream<Integer>> with the result of an HTTP request transformed in a sequence of temperatures in Celsius, then we can perform both queries in the following way:
Flux<Integer> cache = Flux.fromStream(temps::join).cache();
cache.distinct().count().subscribe(out::println);
cache.reduce(Integer::max).subscribe(out::println);
This solution avoids: 1) IllegalStateException on further traversals of this sequence; 2) a first traversal to collect items in a cache.

Remove elements from ArrayList

I have to remove elements from ArrayList, but I have not gone through it. Elements which I have to remove are also available in ArrayList. In short, I have to remove one Array List from another Array List. e.g. Suppose
ArrayList<String> arr1= new ArrayList<String>();
ArrayList<String> arr2 = new ArrayList<String>();
arr1.add("1");
arr1.add("2");
arr1.add("3");
arr2.add("2");
arr2.add("4");
Now, I have to remove elements which are in arr2 from arr1. So, that I have final answer as 1 and 3.
What needs to be done?

Read Remove Common Elements in Two Lists Java
Use below code
List<String> resultArrayList = new ArrayList<String>(arr1);
resultArrayList.removeAll(arr2);
Or can be done by
arr1.removeAll(arr2)
After SO comments
I used the following code
ArrayList<String> arr1= new ArrayList<String>();
ArrayList<String> arr2 = new ArrayList<String>();
arr1.add("1");
arr1.add("2");
arr1.add("3");
arr2.add("2");
arr2.add("4");
System.out.println("Before removing---");
System.out.println("Array1 : " + arr1);
System.out.println("Array2 : " + arr2);
System.out.println("Removing common ---");
List<String> resultArrayList = new ArrayList<String>(arr1);
resultArrayList.removeAll(arr2);
System.out.println(resultArrayList);
and getting output as
Before removing---
Array1 : [1, 2, 3]
Array2 : [2, 4]
Removing common ---
[1, 3]
So what is not working at your side?
Read more about How do you remove the overlapping contents of one List from another List?

Take new arr as final sorted array
for(int i=0;i<arr1.size();i++)
{
for(int j=0;j<arr2.size();j++)
if(!arr1.get(i).contains(arr2.get(j)))
{
arr.add(arr1.get(i));
}
}

You can use removeAll() function
/**
* Removes from this list all of its elements that are contained in the
* specified collection.
*
* #param c collection containing elements to be removed from this list
* #return {#code true} if this list changed as a result of the call
* #throws ClassCastException if the class of an element of this list
* is incompatible with the specified collection
* (optional)
* #throws NullPointerException if this list contains a null element and the
* specified collection does not permit null elements
* (optional),
* or if the specified collection is null
* #see Collection#contains(Object)
*/
public boolean removeAll(Collection<?> c) {
return batchRemove(c, false);
}

To remove duplicate of one from other use this
int arr1Size = arr2.size();
int arr2Size = arr2.size();
for (int i = 0; i < arr1Size; i++)
{
for (int j = 0; j < arr2Size; j++)
{
if (arr1.get(i).contains(arr2.get(j)))
{
arr1.remove(i);
}
}
}
System.out.print(arr1);

Ok to make things clear:
if your list is composed of basic elements such as String etc
all you need to do is use
list2.removeAll(list1);
assuming that isnt the case meaning you created a list from custum objects - the above method wont work, that is due to the nature of the item comparison.
it uses the object.equals method which by default checks if this is the same instance of the object in the other list (which it probably isnt)
so in order for this to work you need to overwrite the custom object equals method.
example - test if 2 contacts are the same based on phone number:
public boolean equals(Object o)
{
if (o==null)
{
return false;
}
if (o.getClass()!=this.getClass())
{
return false;
}
Contact c=(Contact)o;
if (c.get_phoneNumber().equals(get_phoneNumber()))
{
return true;
}
return false;
}
now if you use
list2.removeAll(list1);
it will compare the items based on the desired attribute (in the example based on phone number) and will work as planned.